PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Long-Tailed Adaptation of Vision-Language Models

Keliang Chen, Yaxin Hou, Hui Liu, Yuheng Jia

arXiv:2610.07903v1cs.CV

TL;DR

Unsupervised vision-language adaptation typically assumes uniform unlabeled data, while real-world distributions can be long-tailed and expose head-class boundary erosion and confusable-class bias. MARS preserves head boundaries and refines tail and confusable classes; across nine benchmarks, it achieves an average improvement of 4.71 percentage points.

  • Problem

    Existing unsupervised VLM adaptation assumes uniform unlabeled data, while long-tailed settings expose head-class boundary erosion and confusable-class bias.

    Source: §1 · pp. 1–2

  • Method

    MARS uses zero-shot visual structure to preserve head-class boundaries, then applies margin-aware self-refinement to improve tail and confusable classes.

    Source: §1 · p. 2

  • Results

    Across nine benchmarks with varying imbalance ratios, MARS achieves an average improvement of 4.71 percentage points.

    Source: §5 · p. 10

  • Takeaways & Limitations

    In the evaluated ULTA settings, MARS maintains head-class boundaries while enhancing tail- and confusable-class discriminability.

    Source: §1 · p. 2, §5 · p. 10

    Class counts and the imbalance ratio are not accessible in practice for the unlabeled dataset.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Adapting vision-language models to downstream tasks has achieved remarkable success by leveraging pseudo-labels generated from unlabeled data. Existing methods typically assume a uniform unlabeled data distribution, and thus the resulting pseudo-label distribution is likewise uniform. However, real-world data distributions are often long-tailed. To tackle this, we formalize a new scenario termed Unsupervised Long-Tailed Adaptation (ULTA). Under this scenario, existing methods exhibit a contrasting phenomenon: head-class performance drops sharply, which is distinct from supervised long-tailed learning where tail classes suffer the most. In particular, we uncover that the distributional mismatch not only erodes head-class boundaries, but also pushes head samples into confusable classes, reinforcing the model's inherent bias. To address these issues, we propose a novel model called Margin-Aware Refinement with Structural alignment (MARS). Specifically, we mitigate head-class boundary erosion via Boundary-Preserving Alignment, which takes the zero-shot VLM as a fixed visual reference to suppress probability increases that lack visual support in the training targets. Building upon this, we introduce Margin-aware Self-Refinement, which employs a dynamic adjustment strategy to refine tail and confusable classes while preventing prediction bias. Extensive experiments on nine benchmark datasets demonstrate that MARS outperforms state-of-the-art methods, achieving an average accuracy improvement of 4.71 percentage points.

1 INTRODUCTION

The introduction identifies unsupervised adaptation to long-tailed data as a distinct challenge: uniform pseudo-label assumptions can erode head-class boundaries and intensify confusable-class bias. MARS addresses these failures by preserving head-class boundaries before refining tail and confusable classes.

1 INTRODUCTION

  • Unlabeled data offers an alternative to labeled fine-tuning, for which sufficient annotations can be impractical in specialized scenarios such as medical imaging.Vision-language models can be adapted by exploiting unlabeled data.Source: §1 · p. 1
  • Existing unsupervised methods assume uniform pseudo-labels, but long-tailed data causes sharp head-class degradation, unlike supervised long-tailed learning, where tail classes typically suffer most.The uniform assumption can misclassify head samples as tail classes; fitting these false samples erodes head-class boundaries, while fine-tuning also exacerbates zero-shot bias toward confusable classes.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • MARS combines Boundary-Preserving Alignment, which uses zero-shot CLIP’s visual structure to suppress unsupported probability increases, with Margin-aware Self-Refinement for tail and confusable classes.Self-refinement dynamically balances teacher guidance and pseudo-label supervision while preserving the well-learned decision boundary.Source: §1 · p. 2
  • The paper introduces Unsupervised Long-Tailed Adaptation as a realistic, under-explored setting for adapting vision-language models to long-tailed unlabeled data.The authors state that, to their knowledge, this is the first study of the setting.Source: §1 · p. 2
  • Across nine benchmarks, MARS achieves state-of-the-art performance with an average improvement of 4.71 percentage points.The experiments cover different imbalance ratios.Source: §1 · p. 2

2 RELATED WORK

Related work reviews prompt-based VLM adaptation and approaches that mine supervision from unlabeled data. It notes that labeling dependence limits prompt-learning methods in unsupervised settings, while a uniform prior limits unlabeled-data methods under long-tailed distributions.

2.1 PROMPT LEARNING IN VLMS

  • Prompt-learning methods adapt VLMs with learnable prompts, including text-context vectors in CoOp, visual prompts in VPT, and multimodal prompting in MaPLe.These methods rely heavily on labeled data, constraining their use in unsupervised scenarios.Source: §2.1 · p. 3

2.2 LEARNING FROM UNLABELED DATA FOR VLMS

  • Unlabeled-data methods improve pseudo-labels or mine other signals, but their reliance on a uniform prior limits applicability to long-tailed data.Examples include confidence filtering in UPL [Huang et al., 2022], iterative top-k selection in GRIP [Menghini et al., 2023], and marginal-entropy regularization in UEO [Liang et al., 2024].Source: §2.2 · p. 3

3 METHODOLOGY

The section formulates ULTA as adaptation to unlabeled long-tailed data and develops a preserve-before-refine pipeline: zero-shot visual structure protects head-class boundaries before tail and confusable classes are refined.

3.1 PROBLEM FORMULATION AND CHALLENGES

  • Existing methods’ uniform-prior pseudo-labeling can cause severe boundary erosion and prediction bias in unlabeled long-tailed adaptation.The imbalance ratio can be defined from class counts, but those counts are not accessible in practice.Source: §3.1 · p. 3

3.2 ANALYSIS OF MISALIGNMENT

  • Fitting a head sample’s incorrect tail pseudo-label shifts its visual feature toward the tail embedding and can redirect attention from the foreground to background noise.Before fitting, the model may still assign high confidence to the correct head class.Source: §3.2 · pp. 3–4

3.3 BOUNDARY-PRESERVING ALIGNMENT

  • BPA uses zero-shot CLIP as a fixed visual reference, class-level visual anchors, and support comparisons to correct unsupported probability increases in training targets.Supported increases, including those correcting zero-shot errors, are preserved.Source: §3.3 · pp. 4–5
  • BPA restores attention to foreground objects and keeps head-class accuracy stable as imbalance increases.Figure 2 contrasts background distraction with restored foreground attention; Figure 3(a) shows head-class accuracy across imbalance ratios.Source: Figure 2 · p. 4, §3.3 · p. 5, Figure 3 · p. 6

3.4 MARGIN-AWARE SELF-REFINEMENT

  • MSR refines with a frozen aligned teacher and student, balancing teacher-preserving KL loss against pseudo-label cross-entropy.The gating factor α balances preserving established concepts with learning new features.Source: §3.4 · p. 5
  • MSR raises teacher guidance for frequent head-class predictions and lowers it for tail or confusable classes, then uses cumulative margin consistency to make α sample-specific.Correctly pseudo-labeled samples mostly gain margin, whereas incorrectly labeled samples maintain negative margins.Source: §3.4 · pp. 5–6
  • On OxfordPets, the complete MARS pipeline retains overall accuracy as the long-tailed distribution becomes more severe.The figure also reports that BPA preserves head-class accuracy as the imbalance ratio increases.Source: Figure 3 · p. 6, §3.4 · p. 6

3.5 MODEL TRAINING AND PREDICTION

  • Training learns prompts and adapters while keeping pretrained image and text encoders frozen; at test time, the teacher adapter is discarded and labels use cosine similarity.Predictions compare refined-encoder visual features with precomputed textual embeddings.Source: §3.5 · p. 6

4 EXPERIMENTS

Across long-tailed settings, MARS outperforms existing methods; its components contribute complementary gains, and analyses report preserved head-class margins and improved tail and confusable-class performance.

4.1 EXPERIMENTAL SETTING

  • Experiments cover nine diverse classification datasets, with long-tailed training splits using imbalance ratios 10, 20, and 50 and seven standardized baselines.The long-tailed splits use an exponential class-frequency decay, with head, medium, and tail groups defined by training frequency.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

  • MARS exceeds the previous state-of-the-art by 4.12, 4.50, and 5.52 pp at imbalance ratios 10, 20, and 50, respectively.The comparison covers nine benchmark datasets and three seeds.Source: Table 1 · p. 7, §4.2 · p. 8

4.3 ABLATION STUDIES

  • BPA and MSR individually improve over zero-shot CLIP by 11.70 pp and 9.79 pp on average, while their combination boosts accuracy by 20.11 pp.The authors suggest BPA’s larger gain may reflect its mitigation of feature drift and boundary erosion; the full model drops only 0.97 pp from ratio 10 to 50.Source: Table 4 · p. 8, §4.3 · pp. 8–9

4.2 MAIN RESULTS

  • Under the balanced setting, MARS reaches 86.27% average accuracy, 0.30 pp above the leading baseline.The comparison uses Caltech101, Flowers102, and OxfordPets.Source: Table 2 · p. 8, §4.2 · p. 8
  • On OpenCLIP, MARS gains an average 8.79 pp over the best-performing baseline across three imbalance ratios.The reported comparison covers four benchmarks.Source: Table 3 · p. 8, §4.2 · p. 8

4.4 MORE ANALYSES

  • On RESISC45 at γ = 50, CPL raises head-to-tail predictions from 428 to 1,060, while BPA reduces them to 392 and maintains positive margin changes.CPL’s negative head-margin shift is associated with boundary erosion under its uniform constraint.Source: §4.4 · p. 9, Figure 4 · p. 9
  • MSR improves tail and medium accuracy while preserving head performance, and yields higher frequency consistency in confusable classes.The frequency-consistency measure reflects alignment between predicted and ground-truth class distributions.Source: Figure 5 · p. 9, §4.4 · p. 9

4.5 MORE CHALLENGING SETTINGS

  • Across Places-LT, ImageNet-LT, and an iNaturalist2018 subset, MARS achieves the best performance and improves average accuracy by 2.51 pp.The benchmarks use imbalance ratios of 996, 256, and 500, respectively.Source: Table 5 · p. 9, §4.5 · p. 9

5 CONCLUSION

MARS addresses ULTA, where uniform constraints erode head-class boundaries and reinforce prediction bias in confusable classes. Across nine benchmarks, it improves average accuracy by 4.71 percentage points.

5 CONCLUSION

  • Across nine benchmarks, MARS achieves an average improvement of 4.71 percentage points.Source: §5 · p. 10
  • Uniform constraints in ULTA erode head-class boundaries while reinforcing prediction bias in confusable classes.Source: §5 · p. 10
  • MARS combines Boundary-Preserving Alignment to maintain head-class boundaries with Margin-aware Self-Refinement to enhance tail and confusable-class discriminability.Source: §5 · p. 10

A TRAINING PROCESS OF MARS

MARS trains through Boundary-Preserving Alignment followed by Margin-aware Self-Refinement. The algorithm caches zero-shot references, aligns targets, then freezes teacher predictions and refines the student.

A TRAINING PROCESS OF MARS

  • MARS training proceeds in two phases: Boundary-Preserving Alignment, then Margin-aware Self-Refinement.The algorithm describes the complete training procedure.Source: A TRAINING PROCESS OF MARS · p. 14
  • Before alignment, the procedure caches zero-shot features and weak-view predictions and constructs class anchors.Source: A TRAINING PROCESS OF MARS · p. 14
  • During alignment, frozen visual features and current predictions determine visual conditions, pairwise coefficients, and targets used to update prompts and the teacher adapter.Source: A TRAINING PROCESS OF MARS · p. 14
  • For refinement, teacher predictions, visual prompts, and the teacher adapter are frozen while the student adapter and textual prompts are updated.The procedure computes a class prior and gating factor before updating the refinement loss.Source: A TRAINING PROCESS OF MARS · p. 14

B DATASET DETAILS

The benchmark evaluation spans diverse domains and includes both balanced and imbalanced test sets. Table 6 reports dataset class counts and training and test sample counts.

B DATASET DETAILS

C IMPLEMENTATION DETAILS

The implementation standardizes baseline comparisons around OpenAI CLIP ViT-B/32 and uses fixed MARS training settings. A sensitivity analysis varies β on Caltech101 and RESISC45.

C IMPLEMENTATION DETAILS

  • All baselines use OpenAI CLIP ViT-B/32, the same training data and split, and the same evaluation metrics.Baselines were rerun from their official public codebases.Source: C IMPLEMENTATION DETAILS · p. 14
  • MARS uses 10 alignment epochs and 30 student-training epochs, with β = 1 and momentum coefficient λ = 0.95 across datasets.Stage 1 uses batch size 64 and SGD; Stage 2 trains the student model.Source: C IMPLEMENTATION DETAILS · p. 14
  • Experiments ran on a single NVIDIA RTX 4090 GPU using PyTorch v2.8.0.Source: C IMPLEMENTATION DETAILS · p. 15

D.1 SENSITIVITY OF HYPERPARAMETERS

  • The sensitivity analysis evaluates β on Caltech101 and RESISC45; Figure 6 identifies the left and right panels as those datasets, respectively.Source: §D.1 · p. 15, Figure 6 · p. 15

D.2 STATISTICAL SIGNIFICANCE ANALYSIS

The supplementary analyses test MARS across VLMs and visual backbones and assess the statistical significance of its improvements. The results report advantages across the tested settings.

D.2 STATISTICAL SIGNIFICANCE ANALYSIS

  • MARS achieves significant improvements in 84.7% of paired comparisons (160 of 189) against seven baselines at the 0.05 significance level.The paired t-tests use three corresponding seeds across different imbalance ratios.Source: Table 7 · p. 15, §D.2 · p. 15

D.3 COMPARISON WITH DIFFERENT VLMS

  • MARS maintains a performance advantage on OpenCLIP and SigLIP; on RESISC45 with SigLIP at imbalance ratio 50, it exceeds the best baseline by 5.10 pp.The comparison also covers OpenCLIP at imbalance ratios 10 and 50.Source: §D.3 · p. 15, Table 9 · p. 16, Table 8 · p. 16

D.4 COMPARISON WITH DIFFERENT VISUAL BACKBONES

  • With ViT-B/16, MARS achieves 74.41% average accuracy, outperforming the strongest baseline by 3.82 pp across four datasets.The comparison evaluates imbalance ratios of 10, 20, and 50.Source: Table 10 · p. 16, §D.4 · p. 16

D.5 FINE-GRAINED RESULTS ACROSS HEAD, MEDIUM, AND TAIL GROUPS

The fine-grained comparison reports results for head, medium, and tail groups on RESISC45 and EuroSAT at imbalance ratio 50. MARS improves head-class accuracy while maintaining medium- and tail-class performance.

D.5 FINE-GRAINED RESULTS ACROSS HEAD, MEDIUM, AND TAIL GROUPS

  • On RESISC45, MARS improves head accuracy by 18.65 pp over UEO and 32.57 pp over CAP.The comparison reports test accuracy across head, medium, and tail groups on RESISC45 and EuroSAT at imbalance ratio 50.Source: §D.5 · p. 17, Table 11 · p. 17
  • MARS improves overall accuracy and strongly restores head-class performance while maintaining medium- and tail-class performance.These group-level results cover RESISC45 and EuroSAT at imbalance ratio 50.Source: §D.5 · pp. 16–17, Table 11 · p. 17
  • Table 11 compares test accuracy across head, medium, and tail groups on RESISC45 and EuroSAT at imbalance ratio 50.Source: Table 11 · p. 17

D.6 HEAD-TO-TAIL MISASSIGNMENT ANALYSIS

MARS improves base-to-new generalization under long-tailed adaptation and addresses head-to-tail misclassification through BPA. The analyses report restored head accuracy and positive margin changes for affected samples.

D.6 HEAD-TO-TAIL MISASSIGNMENT ANALYSIS

  • On RESISC45, CPL lowers head accuracy from 69.20% to 38.69% and raises H-to-T errors from 428 to 1,060; BPA restores 72.89% head accuracy and reduces H-to-T errors to 392 without sacrificing tail accuracy.OxfordPets shows the same pattern.Source: Table 12 · p. 17, §D.6 · p. 17
  • For head samples CPL predicts as tail, CPL yields strongly negative margin changes, whereas BPA turns them positive and recovers the correct label for 94.12% on Caltech101 and 61.11% on Food101.These analyses use an imbalance ratio of 50 and measure margin changes relative to zero-shot CLIP.Source: Table 13 · p. 17, §D.6 · p. 17

D.7 BASE-TO-NEW GENERALIZATION

  • On RESISC45, MARS raises the best prior harmonic mean from 58.92% to 62.16%, a gain of 3.24 pp, and outperforms prior methods on both datasets.The base-to-new evaluation uses a 1:1 base/new split and evaluates with the harmonic mean.Source: §D.7 · pp. 17–18, Table 14 · p. 18

Related papers

Loading 2610.07903v1…