PaperlayerView exact revision on arXiv

Source-linked AI summary

Semi-Supervised Adaptation of Vision-Language Models for Image Classification

Mohamed L. Mekhalfi, Mohamad M. Al Rahhal, Yakoub Bazi, Salah E. Khenfer, Mingdeng Shi, Hua Zou, Mansour Zuair

arXiv:2608.25485v1cs.CV

TL;DR

Remote-sensing domain gaps and scarce expert labels limit CLIP adaptation, while noisy pseudo-labeling can reinforce class imbalance. SE-CLIP combines parameter-efficient adaptation with warm-up initialization, recursive high-confidence discovery, and class-balanced mining; it outperforms representative semi-supervised methods on UCM and NWPU, but remains limited in fine-grained scenes requiring expressive text vocabularies.

  • Problem

    Remote-sensing adaptation is constrained by domain discrepancy, scarce expert annotations, and noisy pseudo-labeling that can reinforce majority-class errors.

    Source: §I · p. 1

  • Method

    SE-CLIP uses parameter-efficient low-rank adaptation, a warm-up on labeled seeds, recursive discovery from unlabeled data, and class-balanced pseudo-label mining.

    Source: §I · pp. 1–2, B. Architectural Adaptation via LoRA · p. 2, D. Recursive Discovery · p. 2

  • Results

    SE-CLIP outperforms representative semi-supervised frameworks, achieving 99.09% accuracy on UCM and 95.07% on NWPU.

    Source: C. Qualitative analysis · p. 4

  • Takeaways & Limitations

    The framework offers a minimal-human-intervention approach for adapting CLIP to complex remote-sensing data distributions.

    Source: §IV · p. 4

    Performance remains limited in fine-grained scenarios with high inter-class structural similarity, and the method requires expressive text vocabularies for discriminative anchors.

    Source: §IV · p. 4

Abstract

from arXiv · show

Vision-language models like CLIP have shown sig- nificant potential in handling natural images, yet their perfor- mance is often limited by the distinct characteristics of satellite imagery. While parameter-efficient adaptation techniques exist, their efficacy is frequently limited by the scarcity of annotated samples. In this letter, we propose Self-Evolutionary CLIP (SE- CLIP), a semi-supervised framework designed for recursive label mining in scene classification. The approach follows a dual-phase pipeline, where an initial warm-up on a few annotated seeds is followed by a recursive discovery phase that iteratively identifies high-confidence samples from unlabeled pools. To maintain the integrity of the evolving support set, we employ a class-balanced selection strategy that prevents the model from being dominated by easily learned categories. Results on the UCM and NWPU benchmarks indicate that SE-CLIP significantly outperforms existing semi-supervised approaches. The framework provides a viable solution for adapting VLMs to the remote sensing domain with minimal human intervention.

I. INTRODUCTION

Remote-sensing imagery creates a domain gap and labeling bottleneck for adapting vision-language models. SE-CLIP addresses these constraints with recursive, class-balanced label discovery from sparse expert-labeled seeds.

  • Satellite imagery differs from general-purpose pre-training through specialized spectral-spatial geometries, limiting VLM performance.Source: §I · p. 1
  • Parameter-efficient adaptation reduces full-fine-tuning costs but remains dependent on high-quality expert annotations.Source: §I · p. 1
  • Specialized geographical labeling is labor-intensive, creating a bottleneck for large-scale remote-sensing monitoring.Source: §I · p. 1
  • Traditional semi-supervised methods risk confirmation bias and majority-class errors when pseudo-labeling noisy satellite imagery.Source: §I · p. 1
  • SE-CLIP replaces rigid thresholding with recursive discovery in CLIP’s semantic space, beginning from sparse expert-labeled seeds and using fixed textual anchors.Source: §I · p. 1
  • A class-balanced migration strategy expands the support set evenly, preventing easy classes from dominating label mining.Source: §I · p. 2

II. METHODOLOGY

SE-CLIP formulates remote-sensing classification with a small labeled support set and a much larger unlabeled pool. It recursively adds high-confidence discoveries to exploit unlabeled structural information.

A. Problem Formulation

B. Architectural Adaptation via LoRA

SE-CLIP adapts CLIP with LoRA by inserting trainable low-rank bypass matrices into an otherwise frozen Transformer. The low-rank constraint targets relevant features while limiting overfitting to sparse labels.

B. Architectural Adaptation via LoRA

C. Semantic Anchoring and Alignment

SE-CLIP replaces a traditional linear classifier with fixed text-derived anchors and trains visual representations to align with the corresponding class anchor. Cross-entropy uses temperature-scaled cosine similarities, retaining CLIP’s pretrained calibration.

C. Semantic Anchoring and Alignment

D. Recursive Discovery

SE-CLIP recursively expands its support set after warming up on expert-labeled seeds. It ranks unlabeled samples by similarity to class text anchors, migrates a fixed budget per class, and maintains balanced growth.

  • The warm-up phase updates LoRA parameters exclusively on expert-labeled seeds to stabilize training gradients.Source: D. Recursive Discovery · p. 2
  • During discovery, samples in the unlabeled pool are ranked by similarity to each class’s text anchor.Source: D. Recursive Discovery · p. 2
  • A fixed budget of k samples per class is migrated into the support set and never re-evaluated.Source: D. Recursive Discovery · p. 2
  • Discovered samples are appended recursively to the labeled set, enabling continued evolution of the adapted visual encoder.Source: D. Recursive Discovery · p. 2
  • Uniform per-class migration produces balanced, high-purity support-set growth and prevents dominance by frequent or easily discriminable classes.Source: D. Recursive Discovery · p. 2
  • The pipeline synchronizes visual-textual representations on labeled seeds before mining similar unlabeled samples for support-set expansion.Source: Figure 2 · p. 3

A. Datasets and Settings

The evaluation uses UCM and NWPU remote-sensing benchmarks with different dataset sizes and class counts. Training begins with five labeled seeds per class, while 20% of each dataset is reserved for testing.

B. Quantitative Analysis

SE-CLIP’s performance depends on mining-budget and adaptation-capacity choices, while recursive, class-balanced discovery improves stability and accuracy. The framework also adapts CLIP efficiently, updating a small parameter fraction and discarding mining modules at deployment.

Mining budget

  • 98.81% accuracy is achieved on UCM at k = 1, while NWPU accuracy rises from 88.56% at k = 1 to 94.92% at k = 5.Increasing NWPU’s budget further to k = 7 slightly reduces accuracy to 94.77%.Source: B. Quantitative Analysis · p. 3

LoRA rank

  • Increasing LoRA rank r from 2 to 16 improves accuracy from 98.71% to 99.09% on UCM and from 94.41% to 95.07% on NWPU.The authors attribute the gain to greater low-rank capacity for adapting to remote-sensing textures and class boundaries.Source: B. Quantitative Analysis · p. 3

Recursive discovery

  • The full recursive discovery mechanism is necessary: warm-up alone causes significant performance drops on both benchmarks.Warm-up stabilizes initialization, but continuous recursive optimization discovers and assimilates additional samples.Source: B. Quantitative Analysis · p. 3

Balanced mining

  • Per-class balanced mining outperforms global mining, whose unconstrained selection causes catastrophic degradation and elevated standard deviations on both datasets.Uniform category distribution is reported as essential for robust semi-supervised adaptation.Source: B. Quantitative Analysis · p. 3

Stability

  • Local selection maintains cumulative pseudo-label standard deviation at zero across all optimization iterations, unlike global selection’s post-warm-up surge.The comparison covers both UCM and NWPU.Source: B. Quantitative Analysis · p. 3

Computational efficiency

  • LoRA optimizes only 1.16% of the CLIP model volume, while deployment inference takes 10.26 ms per batch of size 16.Recursive label-mining modules are discarded during deployment.Source: B. Quantitative Analysis · p. 3

C. Qualitative analysis

SE-CLIP improves performance through recursive label mining, with convergence occurring earlier on UCM than on the larger NWPU benchmark. It also achieves higher classification accuracy than representative semi-supervised baselines on both datasets.

Optimization dynamics

  • After the 10-iteration warm-up, top-k label mining produces a smooth accuracy increase on both benchmarks.UCM stabilizes around iteration 60, while NWPU reaches its steady-state plateau between iterations 160 and 170.Source: C. Qualitative analysis · p. 4

Benchmark comparison

IV. CONCLUSION

SE-CLIP adapts CLIP for semi-supervised remote-sensing scene classification using parameter-efficient fine-tuning and balanced pseudo-label mining guided by fixed textual anchors. The framework remains constrained by the expressiveness of its text vocabulary and by fine-grained classes with high structural similarity.

Conclusion

  • SE-CLIP leverages CLIP’s pretrained capabilities through parameter-efficient fine-tuning for semi-supervised scene classification.Source: §IV · p. 4
  • Balanced pseudo-label mining based on fixed textual anchors supports adaptation to complex remote-sensing data distributions.Source: §IV · p. 4

Limitations

  • The framework requires expressive text vocabularies to construct discriminative anchors.Source: §IV · p. 4
  • Performance remains limited for fine-grained scenarios with high inter-class structural similarity when semantic descriptions cannot fully resolve visual overlap.Source: §IV · p. 4

Related papers

Loading 2608.25485v1…