Source-linked AI summary
Semi-Supervised Adaptation of Vision-Language Models for Image Classification
Mohamed L. Mekhalfi, Mohamad M. Al Rahhal, Yakoub Bazi, Salah E. Khenfer, Mingdeng Shi, Hua Zou, Mansour Zuair
TL;DR
Remote-sensing domain gaps and scarce expert labels limit CLIP adaptation, while noisy pseudo-labeling can reinforce class imbalance. SE-CLIP combines parameter-efficient adaptation with warm-up initialization, recursive high-confidence discovery, and class-balanced mining; it outperforms representative semi-supervised methods on UCM and NWPU, but remains limited in fine-grained scenes requiring expressive text vocabularies.
Problem
Remote-sensing adaptation is constrained by domain discrepancy, scarce expert annotations, and noisy pseudo-labeling that can reinforce majority-class errors.
Method
SE-CLIP uses parameter-efficient low-rank adaptation, a warm-up on labeled seeds, recursive discovery from unlabeled data, and class-balanced pseudo-label mining.
Results
SE-CLIP outperforms representative semi-supervised frameworks, achieving 99.09% accuracy on UCM and 95.07% on NWPU.
Takeaways & Limitations
The framework offers a minimal-human-intervention approach for adapting CLIP to complex remote-sensing data distributions.
Takeaways & Limitations
Performance remains limited in fine-grained scenarios with high inter-class structural similarity, and the method requires expressive text vocabularies for discriminative anchors.
Abstract
from arXiv · showhide
Vision-language models like CLIP have shown sig- nificant potential in handling natural images, yet their perfor- mance is often limited by the distinct characteristics of satellite imagery. While parameter-efficient adaptation techniques exist, their efficacy is frequently limited by the scarcity of annotated samples. In this letter, we propose Self-Evolutionary CLIP (SE- CLIP), a semi-supervised framework designed for recursive label mining in scene classification. The approach follows a dual-phase pipeline, where an initial warm-up on a few annotated seeds is followed by a recursive discovery phase that iteratively identifies high-confidence samples from unlabeled pools. To maintain the integrity of the evolving support set, we employ a class-balanced selection strategy that prevents the model from being dominated by easily learned categories. Results on the UCM and NWPU benchmarks indicate that SE-CLIP significantly outperforms existing semi-supervised approaches. The framework provides a viable solution for adapting VLMs to the remote sensing domain with minimal human intervention.
I. INTRODUCTION
Remote-sensing imagery creates a domain gap and labeling bottleneck for adapting vision-language models. SE-CLIP addresses these constraints with recursive, class-balanced label discovery from sparse expert-labeled seeds.
- Satellite imagery differs from general-purpose pre-training through specialized spectral-spatial geometries, limiting VLM performance.
- Parameter-efficient adaptation reduces full-fine-tuning costs but remains dependent on high-quality expert annotations.
- Specialized geographical labeling is labor-intensive, creating a bottleneck for large-scale remote-sensing monitoring.
- Traditional semi-supervised methods risk confirmation bias and majority-class errors when pseudo-labeling noisy satellite imagery.
- SE-CLIP replaces rigid thresholding with recursive discovery in CLIP’s semantic space, beginning from sparse expert-labeled seeds and using fixed textual anchors.
- A class-balanced migration strategy expands the support set evenly, preventing easy classes from dominating label mining.
II. METHODOLOGY
SE-CLIP formulates remote-sensing classification with a small labeled support set and a much larger unlabeled pool. It recursively adds high-confidence discoveries to exploit unlabeled structural information.
- A. Problem Formulation: The dataset is partitioned into a labeled support set L and a significantly larger unlabeled pool U.
- A. Problem Formulation: Each labeled sample pairs an image x_i with a categorical label y_i among C distinct classes.
- A. Problem Formulation: The support set is much smaller than the unlabeled pool, with N_L ≪ N_U.
- A. Problem Formulation: The objective minimizes empirical risk on L while using structural information in U to improve generalization on unseen test data.
- A. Problem Formulation: SE-CLIP iteratively expands L with high-confidence samples discovered from U through recursive alignment.
B. Architectural Adaptation via LoRA
SE-CLIP adapts CLIP with LoRA by inserting trainable low-rank bypass matrices into an otherwise frozen Transformer. The low-rank constraint targets relevant features while limiting overfitting to sparse labels.
- B. Architectural Adaptation via LoRA: CLIP supplies the visual and textual encoders, while LoRA bridges natural-image pre-training and satellite imagery characteristics.
- B. Architectural Adaptation via LoRA: LoRA injects trainable matrices A and B into frozen Transformer weights to specialize the foundation model efficiently.
- B. Architectural Adaptation via LoRA: The intrinsic rank r is typically much smaller than both matrix dimensions, r ≪ min(d, l).
- B. Architectural Adaptation via LoRA: The low-rank constraint focuses adaptation on relevant features while preventing overfitting to sparse initial labels.
C. Semantic Anchoring and Alignment
SE-CLIP replaces a traditional linear classifier with fixed text-derived anchors and trains visual representations to align with the corresponding class anchor. Cross-entropy uses temperature-scaled cosine similarities, retaining CLIP’s pretrained calibration.
- C. Semantic Anchoring and Alignment: Each class is represented by a fixed textual anchor generated from class-specific prompts through the frozen text encoder.
- C. Semantic Anchoring and Alignment: An adapted visual representation V_i is aligned with the textual anchor T_c corresponding to its class.
- C. Semantic Anchoring and Alignment: Training uses cross-entropy over temperature-scaled cosine similarities between visual representations and class anchors.
- C. Semantic Anchoring and Alignment: The ground-truth textual anchor is T_GT, while e^τ is the constant log-parameterized temperature scale inherited from pretrained CLIP.
D. Recursive Discovery
SE-CLIP recursively expands its support set after warming up on expert-labeled seeds. It ranks unlabeled samples by similarity to class text anchors, migrates a fixed budget per class, and maintains balanced growth.
- The warm-up phase updates LoRA parameters exclusively on expert-labeled seeds to stabilize training gradients.
- During discovery, samples in the unlabeled pool are ranked by similarity to each class’s text anchor.
- A fixed budget of k samples per class is migrated into the support set and never re-evaluated.
- Discovered samples are appended recursively to the labeled set, enabling continued evolution of the adapted visual encoder.
- Uniform per-class migration produces balanced, high-purity support-set growth and prevents dominance by frequent or easily discriminable classes.
- The pipeline synchronizes visual-textual representations on labeled seeds before mining similar unlabeled samples for support-set expansion.
A. Datasets and Settings
The evaluation uses UCM and NWPU remote-sensing benchmarks with different dataset sizes and class counts. Training begins with five labeled seeds per class, while 20% of each dataset is reserved for testing.
- UCM contains 2,100 aerial images across 21 land-use categories, while NWPU contains 31,500 images across 45 classes.
- All scene images are resized to 256 × 256 pixels for evaluation.
- The framework uses OpenAI’s pre-trained ViT-B/32 CLIP model as its core multimodal backbone and is implemented in PyTorch.
- Eighty percent of each dataset is allocated to training, with only five seeds per class initially labeled and the remainder unlabeled.
- The remaining 20% is reserved strictly for testing, with classification accuracy used as the evaluation metric.
B. Quantitative Analysis
SE-CLIP’s performance depends on mining-budget and adaptation-capacity choices, while recursive, class-balanced discovery improves stability and accuracy. The framework also adapts CLIP efficiently, updating a small parameter fraction and discarding mining modules at deployment.
- Mining budget: 98.81% accuracy is achieved on UCM at k = 1, while NWPU accuracy rises from 88.56% at k = 1 to 94.92% at k = 5.Increasing NWPU’s budget further to k = 7 slightly reduces accuracy to 94.77%.
- LoRA rank: Increasing LoRA rank r from 2 to 16 improves accuracy from 98.71% to 99.09% on UCM and from 94.41% to 95.07% on NWPU.The authors attribute the gain to greater low-rank capacity for adapting to remote-sensing textures and class boundaries.
- Recursive discovery: The full recursive discovery mechanism is necessary: warm-up alone causes significant performance drops on both benchmarks.Warm-up stabilizes initialization, but continuous recursive optimization discovers and assimilates additional samples.
- Balanced mining: Per-class balanced mining outperforms global mining, whose unconstrained selection causes catastrophic degradation and elevated standard deviations on both datasets.Uniform category distribution is reported as essential for robust semi-supervised adaptation.
- Stability: Local selection maintains cumulative pseudo-label standard deviation at zero across all optimization iterations, unlike global selection’s post-warm-up surge.The comparison covers both UCM and NWPU.
- Computational efficiency: LoRA optimizes only 1.16% of the CLIP model volume, while deployment inference takes 10.26 ms per batch of size 16.Recursive label-mining modules are discarded during deployment.
C. Qualitative analysis
SE-CLIP improves performance through recursive label mining, with convergence occurring earlier on UCM than on the larger NWPU benchmark. It also achieves higher classification accuracy than representative semi-supervised baselines on both datasets.
- Optimization dynamics: After the 10-iteration warm-up, top-k label mining produces a smooth accuracy increase on both benchmarks.UCM stabilizes around iteration 60, while NWPU reaches its steady-state plateau between iterations 160 and 170.
- Benchmark comparison: 99.09% accuracy on UCM surpasses the strongest baseline, DARP, by 3.22%.
- Benchmark comparison: 95.07% accuracy on NWPU exceeds the strongest baseline by 6.21%.NWPU is described as the more challenging benchmark.
IV. CONCLUSION
SE-CLIP adapts CLIP for semi-supervised remote-sensing scene classification using parameter-efficient fine-tuning and balanced pseudo-label mining guided by fixed textual anchors. The framework remains constrained by the expressiveness of its text vocabulary and by fine-grained classes with high structural similarity.
- Conclusion: SE-CLIP leverages CLIP’s pretrained capabilities through parameter-efficient fine-tuning for semi-supervised scene classification.
- Conclusion: Balanced pseudo-label mining based on fixed textual anchors supports adaptation to complex remote-sensing data distributions.
- Limitations: The framework requires expressive text vocabularies to construct discriminative anchors.
- Limitations: Performance remains limited for fine-grained scenarios with high inter-class structural similarity when semantic descriptions cannot fully resolve visual overlap.