PaperlayerView exact revision on arXiv

Source-linked AI summary

DiscoVL: Unveiling Disentangled C ross-Modal Representation Learning via Orthogonal Adversarial Regularization for V ision-Language Models

Mengping Dong, Jinbao Li, Fei Li

arXiv:2610.11113v1cs.CV

TL;DR

VLM prompt-learning methods can entangle class-specific cues with transferable knowledge, alongside cross-modal inconsistencies and token collapse. DiscoVL separates shared and task-specific representations with cross-modal alignment and orthogonal adversarial constraints. Across 15 benchmarks, it reports consistent performance gains on base and novel classes.

  • Problem

    Existing VLM adaptation methods entangle class-specific discriminative cues with class-agnostic transferable knowledge, while joint-space projections can produce inconsistent cross-modal representations.

    Source: §1 · p. 2

  • Method

    DiscoVL separates adapted representations into shared class-agnostic and task-specific subspaces using cross-modal alignment and orthogonal adversarial constraints.

    Source: §1 · p. 2

  • Results

    Across 15 benchmarks, DiscoVL shows consistent performance gains on both base and novel classes.

    Source: §5 · p. 15

  • Takeaways & Limitations

    Dedicated disentanglement analyses confirm that the separated components are genuinely distinct.

    Source: §1 · p. 3, §5 · p. 15

Abstract

from arXiv · show

Pre-trained vision-language models excel across varied perception tasks, but adapting them to novel downstream settings without sacrificing generalization remains non-trivial. Existing parameter-efficient prompt learning method often yields inconsistent representations and fails to account for semantic distribution shifts. In this work, we present DiscoVL, a disentangled cross-modal representation learning framework that couples orthogonal adversarial regularization with structured cross-modal alignment for vision-language models. To address the insufficient cross-modal interaction, our DiscoVL designs a multi-branch low-rank residual aligner that decomposes representations into subspaces and enables bidirectional cross-modal feedback between visual and textual streams at each layer. Furthermore, while conventional triplet constraints overfit features to class centroids, we design an orthogonal regularization for adversarial triplet loss, which prevents centroid collapse and substantially boosts generalization. Evaluations on 15 benchmarks demonstrate that DiscoVL delivers consistent improvements over state-of-the-art methods for base-to-novel generalization, cross-dataset evaluation, and few-shot learning

1 Introduction

The introduction identifies entangled class-specific and class-agnostic information as a bottleneck in VLM prompt learning. DiscoVL separates these representations and reports consistent gains over prior prompt-learning methods on base-to-novel generalization.

1 Introduction

  • Existing VLM adaptation methods entangle class-specific discriminative cues with class-agnostic transferable knowledge, including methods using prototypes, priors, attributes, or multiple aligners.This entanglement is associated with drift from pretrained features, inconsistent cross-modal representations, and token collapse onto class centroids.Source: §1 · p. 2
  • DiscoVL factorizes adapted representations into a shared class-agnostic subspace for transferable semantics and a task-specific subspace for downstream discriminative cues.DCRA uses multi-branch residual structure, subspace separation, and bidirectional cross-modal feedback to align visual and textual cues across layers.Source: §1 · pp. 2–3
  • OARL applies orthogonal regularization and adversarial triplet objectives to suppress token collapse and preserve pretrained semantic representations.The paper reports evaluations on 15 datasets and dedicated disentanglement analyses confirming the intended subspace separation.Source: §1 · pp. 2–3
  • DiscoVL consistently surpasses prior prompt-learning methods on base-to-novel generalization across 11 benchmarks.Fig. 1 compares harmonic means across 11 diverse datasets.Source: Figure 1 · p. 2, §1 · p. 3

2 Related work

Related work describes VLMs and prompt-learning approaches, then contrasts DiscoVL’s explicit separation of shared and task-specific representations with methods that operate in an entangled feature space.

2 Related work

  • VLM adaptation can entangle class-specific and class-agnostic information, which the paper associates with semantic drift and degraded cross-domain transfer.The section situates this issue in the context of CLIP-like models and other vision-language pipelines.Source: §2 · p. 3
  • Prompt-learning approaches range from learned continuous prompts and branch-aware prompts to constraints, consistency objectives, and target-aware text features.The section names methods including KgCoOp, MaPLe, PromptSRC, CoPrompt, TCP, TextRefiner, and SurPL.Source: §2 · p. 3
  • Unlike approaches that operate in an entangled feature space, DiscoVL factorizes representations into shared and task-specific subspaces and enforces their separation with orthogonal adversarial regularization.Fig. 2 depicts DCRA’s cross-modal alignment and OARL’s constraints against centroid collapse.Source: §2 · p. 4, Figure 2 · p. 4

3 Methodology

DiscoVL adapts CLIP using injected representation tokens, a disentangled cross-modal aligner, and orthogonal adversarial learning. Its design separates task-specific adaptation from pre-trained knowledge while supporting cross-modal interaction.

3.1 Preliminaries

  • DiscoVL injects representation tokens from layer Q into CLIP’s image and text processing, with higher-layer injection intended to preserve low-level pre-training priors.Q marks the first layer receiving the additional context.Source: §3.1 · pp. 4–5

3.2 Disentangled cross-modal representation aligner

  • DCRA combines a globally shared component with gated, weighted low-rank residual branches that adaptively emphasize relevant subspaces for each layer and modality.The branches are designed to capture diverse relationships while keeping parameterization compact.Source: §3.2 · pp. 6–7
  • DCRA also sends textual cues to visual tokens in lower layers and visual summaries to text in upper layers, using a learned gate to balance fusion and modality preservation.The gate varies by layer and prioritizes the modality providing more relevant information at that depth.Source: §3.2 · p. 7

3.3 Orthogonal adversarial representation learning

  • The adversarial triplet objective aligns an image feature with its same-class representation and contrasts it with a different-class image feature; the authors identify centroid collapse as a limit on novel-class generalization.The paper motivates orthogonal regularization as a way to reduce feature redundancy and promote feature independence.Source: §3.3 · p. 8
  • Orthogonal regularization is combined with classification, cosine-consistency, and adversarial-triplet losses, while the CLIP class token retains pre-trained knowledge and alone supports novel-class prediction.Representation tokens are adapted to the target dataset, whereas the class token is used for novel-class prediction.Source: §3.3 · pp. 8–9

4 Experiments

Experiments follow prior benchmarks across four tasks, using 11 diverse datasets and four ImageNet variants for domain generalization. For base-to-novel generalization, DiscoVL improves average Base, Novel, and HM over MMRL and achieves greater HM on 7 of 11 benchmarks.

4.1 Experimental setup

  • Experiments cover four tasks, using 11 diverse-domain datasets and four ImageNet variants for domain generalization.The 11 datasets span generic and fine-grained objects, scenes, actions, textures, and satellite imagery.Source: §4.1 · pp. 9–10
  • The standard setup uses ViT-B/16 CLIP and 16 shots except in few-shot experiments, reporting Base, Novel, and HM averaged over three runs.Training details vary by task and dataset, with few-shot settings of 1, 2, 4, 8, and 16 shots.Source: §4.1 · pp. 10–11

4.2 Base-to-novel generalization

  • DiscoVL improves Base, Novel, and HM over MMRL by 0.30%, 0.45%, and 0.38%, respectively, and has greater HM on 7 of 11 benchmarks.It also outperforms all competitors on average across datasets; on EuroSAT, HM improves by 1.11%.Source: Table 1 · p. 10, §4.2 · p. 11

4.3 Cross-dataset evaluation

Across cross-dataset, domain-generalization, and few-shot evaluations, DiscoVL reports strong results against prior methods. It has the highest average accuracy in cross-dataset evaluation, improves average target accuracy over PromptSRC under domain shift, and leads at reported few-shot settings.

4.3 Cross-dataset evaluation

  • DiscoVL is superior on 6 of 10 cross-dataset benchmarks, with improvements of 0.48%, 0.48%, and 0.23% for CoPrompt, Skip Tuning, and MMRL, respectively.The table caption reports the highest overall average accuracy for DiscoVL.Source: §4.3 · pp. 11–12

4.4 Domain generalization

  • DiscoVL achieves the best ImageNet-A result and improves average target accuracy over PromptSRC by 0.18% across shifted ImageNet variants.Its largest relative improvement is 0.47% on ImageNet-R.Source: §4.4 · p. 12, Table 3 · p. 12

4.5 Few-shot learning

  • 73.48%, 76.63%, and 84.58%: DiscoVL leads at 1-, 2-, and 16-shot, respectively, surpassing previous best methods at 1 and 2 shots.At 16 shots, it exceeds PromptSRC and Skip Tuning by 1.71% and 0.67%, respectively.Source: §4.5 · p. 12, Figure 4 · p. 13

4.6 Ablation study

The ablations report gains from both the multi-branch residual aligner with bidirectional feedback and orthogonal adversarial representation learning. Visualizations show class-wise clustering and distinct shared versus task-specific attention patterns.

4.6 Ablation study

  • DCRA raises HM from 81.20% to 81.32%; adding orthogonal adversarial representation learning increases it to 81.58%.The component study averages over 11 datasets; both modules also improve performance individually.Source: Table 4 · p. 13, §4.6 · p. 13
  • On StanfordCars, adding Lk_atr improves HM by +0.20%, Lk′_atr adds +0.07%, and orthogonal regularization yields the best HM of 78.42%.The authors describe the loss terms as complementary.Source: Table 4 · p. 13, §4.6 · p. 13

4.7 Visualization

  • EuroSAT t-SNE shows clear class-wise clustering; the authors attribute this to cross-modal integration, preserved shared knowledge, and reduced feature redundancy.Different colors denote different classes.Source: §4.7 · p. 14, Figure 5 · p. 14
  • Grad-CAM shows broad shared-branch context and focused task-specific responses; DiscoVL’s attention map is more refined and concentrated than the baseline’s.The task-specific branch pinpoints discriminative local regions, while the shared branch captures class-agnostic context.Source: Figure 6 · p. 14, §4.7 · p. 14

4.8 Training stability and efficiency

On ImageNet, DiscoVL reports lower parameter count and latency than MMRL while slightly improving HM. Training stability varies by dataset and class, with the passage reporting both stronger and weaker standard-deviation comparisons.

4.8 Training stability and efficiency

  • Stability is mixed: DiscoVL is more stable than CoPrompt and MMRL on ImageNet base classes and has the lowest EuroSAT standard deviation, but is less stable than CoPrompt on ImageNet novel classes.The passage also notes increased standard deviation on EuroSAT novel classes compared with other methods.Source: §4.8 · p. 14, Table 5 · p. 15
  • DiscoVL reduces parameters by 55.7% and latency from 6.31 ms to 6.11 ms versus MMRL, while improving HM by 0.26%.Throughput improves and FLOPs decrease slightly; the comparison is on one NVIDIA RTX 3060 GPU.Source: §4.8 · pp. 14–15, Table 6 · p. 15

5 Conclusion

DiscoVL addresses representation entanglement by separating shared and task-specific features, and reports consistent gains across 15 benchmarks. Its analyses find the components distinct and complementary.

5 Conclusion

  • Across 15 benchmarks, DiscoVL reports consistent gains on base and novel classes, with analyses finding distinct, complementary components.The conclusion identifies future work in coupling DCRA with dynamic prompt schedulers and extending orthogonal adversarial learning to sequential or streaming settings.Source: §5 · p. 15
  • DiscoVL identifies representation entanglement as a principal bottleneck and factorizes adapted features into shared and task-specific subspaces.Source: §5 · p. 15
  • DCRA aligns cross-modal cues through multi-branch residual structure and bidirectional feedback, while OARL uses orthogonal adversarial constraints to suppress token collapse.Source: §5 · p. 15

S1 Hyperparameter Analysis

The ablations identify preferred settings for the regularization and branch count, while reporting stable cross-modal fusion weights across layers.

Effectiveness of value γ, α and β

Regularization strategies

Effect of branch E

Gate behavior analysis

S2 Disentanglement Analysis

The analyses report low overlap between shared and task-specific branches and show that the branches contribute differently to base- and novel-class performance.

Separability

Branch intervention

  • On ImageNet, removing the task-specific branch drops base-class accuracy by 6.2% and novel by 1.2%; masking the shared branch changes them by −0.9% and −4.4%.The same trend holds across all three datasets.Source: S2 Disentanglement Analysis · p. 20, Table S6 · p. 20
Loading 2610.11113v1…