LoFT: Parameter-Efficient Fine-Tuning for Long-tailed Semi-Supervised Learning in Open-World Scenarios

Zhiyuan Huang; Jiahao Chen; Bing Su

Source-linked AI summary

LoFT: Parameter-Efficient Fine-Tuning for Long-tailed Semi-Supervised Learning in Open-World Scenarios

Zhiyuan Huang, Jiahao Chen, Bing Su

arXiv:2509.09926v5Sep 12, 2025cs.LGcs.CV

TL;DR

LTSSL must address scarce tail samples, unreliable pseudo-labels, and OOD data that can be forced into in-distribution classes. The paper proposes LoFT through parameter-efficient foundation-model fine-tuning and LoFT-OW with open-world filtering; experiments report superior performance across benchmarks and open-world settings.

Problem
LTSSL models trained from scratch can be overconfident and produce unreliable pseudo-labels, while open-world unlabeled data may contain OOD samples that corrupt learning.
Method
LoFT uses parameter-efficient fine-tuning of foundation models, and LoFT-OW adds built-in OOD filtering for open-world LTSSL.
Results
LoFT achieves competitive performance on traditional LTSSL benchmarks and superior performance in open-world scenarios, including with only 10% of the unlabeled data used by previous works.
Takeaways & Limitations
Foundation-model fine-tuning provides a basis for more reliable pseudo-labeling and robust open-world LTSSL under the reported benchmarks.
Takeaways & Limitations
The theoretical bound assumes the transfer error εtrans is small for foundation models.

Abstract

from arXiv · show

Long-tailed semi-supervised learning (LTSSL) presents a formidable challenge where models must overcome the scarcity of tail samples while mitigating the noise from unreliable pseudo-labels. Most prior LTSSL methods are designed to train models from scratch, which often leads to issues such as overconfidence and low-quality pseudo-labels. To address this problem, we first theoretically prove that utilizing a foundation model significantly reduces the hypothesis complexity, which tightens the generalization bound and in turn minimizes the Balanced Posterior Error (BPE). Furthermore, we demonstrate that the feature compactness of foundation models strictly compresses the acceptance region for outliers, providing a geometric guarantee for robustness. Motivated by these theoretical insights, we extend LTSSL into the foundation model fine-tuning paradigm and propose a novel framework: LoFT (Long-tailed semi-supervised learning via parameter-efficient Fine-Tuning). Furthermore, we explore a more practical setting by investigating semi-supervised learning under open-world conditions, where the unlabeled data may include out-of-distribution (OOD) samples.To handle this problem, we propose LoFT-OW (LoFT under Open-World scenarios) to improve the discriminative ability. Experimental results on multiple benchmarks demonstrate that our method achieves superior performance. Code is available: https://github.com/games-liker/LoFT

1. Introduction

LTSSL addresses long-tailed data by using unlabeled samples, but training from scratch can produce overconfident predictions and unreliable pseudo-labels, especially when unlabeled data includes OOD samples. LoFT uses parameter-efficient foundation-model fine-tuning, while LoFT-OW adds open-world filtering and achieves superior performance in challenging settings.

1. Introduction: Training LTSSL models from scratch can cause overconfidence and unreliable pseudo-labels, particularly for tail classes.These issues motivate using foundation models rather than relying only on early predictions from scratch-trained models.
1. Introduction: LoFT leverages foundation models through parameter-efficient fine-tuning to produce well-calibrated, high-quality pseudo-labels.The framework is motivated by theoretical insights and empirical observations about foundation-model generalization.
1. Introduction: LoFT-OW extends LTSSL to open-world scenarios where unlabeled data may contain OOD samples and filters irrelevant samples with a built-in detection mechanism.This is intended to improve robustness and representation learning under diverse real-world data conditions.
1. Introduction: LoFT achieves competitive performance on CIFAR-LT and ImageNet127, while outperforming previous methods in open-world scenarios using only 10% of the unlabeled data.The reported experiments cover both traditional LTSSL benchmarks and more challenging open-world settings.

2. Related Work

Related work studies long-tailed semi-supervised learning, foundation-model transfer, and confidence calibration. These directions motivate combining unlabeled-data utilization with pretrained representations and better-calibrated confidence.

2. Related Work: LTSSL methods generate pseudo-labels from unlabeled data, with CReST using class-rebalancing self-training to improve long-tailed performance.CReST builds on the observation that imbalanced models can still produce high-precision pseudo-labels for tail classes.
2. Related Work: Foundation models show strong generalization across downstream tasks, but effectively leveraging them for LTSSL remains underexplored.LoFT is positioned as a framework for integrating foundation-model capabilities into LTSSL.
2. Related Work: Confidence calibration aligns predicted confidence with accuracy and is relevant to safety measurement and OOD detection, while modern CNNs tend to be overconfident under long-tailed distributions.Prior work addresses this issue with techniques including mixup pre-training, label-aware smoothing, and batch normalization.

3. Theoretical Motivation and Observations

The paper argues that PEFT reduces hypothesis complexity, tightening generalization and lowering worst-case class risk, while compact foundation-model features improve OOD rejection. Calibration and OOD experiments provide empirical support for these theoretical motivations.

3. Theoretical Motivation and Observations: Restricting the hypothesis space through PEFT compensates for scarce tail-class samples by tightening the generalization bound.The framework compares a broad scratch-trained hypothesis space with a smaller PEFT subspace conditioned on pretrained features.
3. Theoretical Motivation and Observations: PEFT is theoretically guaranteed to produce a lower upper bound on Balanced Posterior Error than training from scratch when transfer error is negligible relative to the complexity gap.The argument connects reduced hypothesis complexity to lower worst-case class-conditional risk and more reliable pseudo-labels.
3. Theoretical Motivation and Observations: Feature compactness in foundation models compresses spherical-cap acceptance regions, reducing the probability that OOD samples are accepted as in-distribution.The geometric argument links tighter feature clusters to stronger rejection capability, with confidence thresholding used to filter OOD noise.
3. Theoretical Motivation and Observations: Calibration experiments compare confidence with accuracy across Many, Medium, and Few classes on ImageNet-LT and Places365-LT, supporting reduced overconfidence with PEFT.The reliability diagrams use ECE, where lower values indicate better confidence-accuracy alignment.
3. Theoretical Motivation and Observations: Open-world experiments evaluate MSP-based OOD detection on multiple datasets and compare fine-tuned foundation models with OE and OCL baselines.The setup tests OOD robustness after fine-tuning on CIFAR-100-LT.

4. Method

LoFT combines parameter-efficient fine-tuning with confidence-aware self-training, while LoFT-OW adds OOD filtering for unlabeled data collected in open-world settings.

4.1. LoFT: LoFT uses a confidence-aware self-training strategy that assigns hard pseudo-labels to high-confidence samples and soft pseudo-labels to low-confidence samples.The method applies hard supervision to reliable predictions and uses the full probability distribution to represent uncertainty for less-confident samples.
4.1. LoFT: LoFT optimizes a supervised classification loss together with an unlabeled-data regularization loss.The supervised objective uses Logit Adjustment, while the unlabeled objective follows weak-to-strong augmentation.
4.2. LoFT-OW (LoFT under Open-World scenarios): LoFT-OW addresses open-world LTSSL by detecting and filtering OOD samples that may enter unlabeled data from unconstrained sources.Such samples can introduce misleading supervision if they are not removed during training.
4.2. LoFT-OW (LoFT under Open-World scenarios): LoFT-OW uses two-stage filtering: zero-shot foundation-model confidence filtering first, followed by filtering with the fine-tuned model’s OOD detection capability.The first stage retains samples above a high-confidence threshold, producing a smaller pseudo-labeled subset for subsequent fine-tuning.

5. Experiments

Experiments evaluate LoFT and LoFT-OW on long-tailed benchmarks, including conventional and open-world settings with OOD data. LoFT outperforms PEFT and prior methods, while LoFT-OW remains competitive with less data and surpasses baselines on ImageNet-127.

Experimental Setup: The evaluation uses CIFAR-100-LT and ImageNet-127, with COCO mixed into the unlabeled pool to simulate open-world OOD data.Experiments compare against FixMatch, ACR variants, TCBC, CPE, and CCL using CLIP and OpenCLIP backbones.
Results on LoFT: LoFT outperforms PEFT across CIFAR-100-LT settings with CLIP and OpenCLIP, reaching up to 83.2% accuracy.Performance remains robust under consistent, uniform, and reversed unlabeled distributions.
Results on LoFT: LoFT reaches 73.3% with CLIP and 73.9% with OpenCLIP on ImageNet-127, exceeding FixMatch+CCL at 67.8%.The comparison supports LoFT’s performance on a larger-scale long-tailed benchmark.
Sensitivity Analysis: LoFT-OW assigns high-confidence pseudo-labels to in-distribution samples and low confidence to uninformative OOD samples.The visualization illustrates the method’s built-in OOD filtering behavior.
Results on LoFT-OW: LoFT-OW remains competitive on CIFAR-100-LT despite OOD-induced distribution shifts and outperforms all baselines, including LoFT, on ImageNet-127.The open-world method uses fewer training iterations and less data.
Sensitivity Analysis: Accuracy rises from 75.6% at c_ood = 0.1 to 76.5% at 0.6, then falls to 75.2% at 0.7.The reported sensitivity study identifies a moderate OOD cutoff as the best balance between filtering and retaining unlabeled data.

6. Conclusion

The paper concludes that parameter-efficient fine-tuning of foundation models addresses key LTSSL limitations and supports robust open-world learning. LoFT-OW combines these insights with dual-stage filtering for OOD-containing unlabeled data.

6. Conclusion: LoFT uses foundation-model fine-tuning to reduce balanced posterior error and enforce feature compactness for tighter OOD acceptance regions.The conclusion frames these theoretical properties as the basis for the proposed framework.
6. Conclusion: LoFT-OW applies dual-stage filtering to improve robustness in open-world long-tailed semi-supervised learning.The framework is presented as achieving strong performance and robustness in the reported experiments.

A.1.1. PRELIMINARIES AND DEFINITIONS

The appendix formulates class-conditional generalization bounds using Rademacher complexity and contrasts training from scratch with PEFT. PEFT reduces effective hypothesis complexity, tightening the bound even when tail-class sample sizes are small.

Preliminaries and Definitions: The appendix defines risk over an input space X and label space Y using hypotheses h: X → R^K and bounded losses.Rademacher complexity is used to analyze generalization error.
PEFT Hypothesis Space: PEFT freezes foundation-model backbone weights and optimizes only a small parameter set, yielding much lower effective Rademacher complexity than training from scratch.The appendix models the PEFT hypothesis space as conceptually contained within, or substantially less complex than, the scratch-trained space.
Class-Conditional Bound: For tail classes, small sample sizes and the large capacity of scratch-trained networks make the class-conditional bound vacuously loose.The complexity term is large for deep networks, while the tail-class sample-size term becomes dominant.
PEFT Generalization Bound: With PEFT, the reduced complexity numerator keeps the generalization bound tight even when the tail-class sample size is small.The restricted search space may introduce a transfer or approximation gap relative to unrestricted training.

A.2. BPE Analysis (Proposition 3.2)

The analysis argues that parameter-efficient fine-tuning with foundation-model features lowers Balanced Posterior Error and geometrically reduces false acceptance of OOD samples. Compact feature distributions yield tighter class acceptance regions than scratch-trained models.

BPE comparison: BPE(hpeft) < BPE(hscr) because parameter-efficient fine-tuning keeps hypothesis complexity small, preserving low worst-case risk for tail classes.Scratch training has loose tail-class bounds and high expected risk, whereas robust foundation features keep the upper bound low.
BPE comparison: Scratch-trained models have loose tail-class decision boundaries, producing large angular acceptance regions because of limited negative-sampling constraints.The resulting worst-case posterior error is determined by tail classes.
Geometric robustness: Foundation-model features remain compact after PEFT, with small angular spread that produces tighter class-specific acceptance regions.The analysis models each class with a spherical cap around its prototype and class threshold.
Geometric robustness: A smaller acceptance angle, corresponding to a larger cosine threshold, exponentially reduces the probability that uniformly distributed OOD samples are falsely accepted.The OOD false-acceptance probability is modeled as the spherical-cap area divided by the sphere area.
Geometric robustness: The analysis concludes that PEFT feature compactness inherently rejects OOD noise by compressing the geometric acceptance region.This provides the stated geometric robustness guarantee for open-world settings.

C. Experimental Details

Experiments fine-tune CLIP and OpenCLIP backbones with AdaptFormer modules using SGD and cosine annealing. Settings vary across CIFAR100-LT and ImageNet-127, with ImageNet-127 using only 1% of training images.

Backbones and optimization: LoFT fine-tunes CLIP and OpenCLIP with AdaptFormer modules inserted into every Transformer block, using SGD with cosine-annealed learning rates.The initial learning rate is 0.01.
CIFAR100-LT: CIFAR100-LT uses 1,024 optimization steps, with learning-rate updates every 32 steps under cosine annealing.The setting is tailored to the relatively small CIFAR100-LT dataset.
Training settings: The SSL configuration uses λ1 = 3.0, λ2 = 0.0, and confidence threshold cu = 0.6, while SSL-OW uses λ1 = 2.0, λ2 = 1.0, cu = 0.95, and cood = 0.6.These settings specify separate standard and open-world semi-supervised learning regimes.
ImageNet-127: ImageNet-127 fine-tunes on only 1% of training images and still outperforms scratch-trained baselines using 10%, with 10,000 total optimization steps.The learning rate is updated every 100 steps via cosine annealing.

D. Additional Confidence Calibration Results

LoFT uses foundation-model confidence estimates to filter OOD samples before semi-supervised training. ECE-based analysis finds these zero-shot confidence estimates highly accurate across several datasets.

OOD filtering: LoFT first filters OOD samples from the unlabeled dataset using the foundation model’s confidence estimates, then visualizes confidence–accuracy curves across datasets for validation.The curves are presented as additional evidence for the reliability of the zero-shot scores.
Confidence calibration: Foundation-model confidence estimates remain highly accurate even in the zero-shot setting according to Expected Calibration Error.This supports using zero-shot confidence scores to guide OOD filtering.

E. Computational Complexity

LoFT improves computational efficiency through faster convergence and parameter-efficient updates. It reaches comparable performance with fewer iterations and updates a small trainable subset of the foundation model.

Overall efficiency: The method is reported to provide significantly superior efficiency in training costs and resource utilization compared with traditional semi-supervised learning frameworks.The detailed efficiency evidence concerns convergence time, update steps, and trainable parameter count.
Training efficiency: LoFT reaches optimal performance in 10,000 iterations and about 4 hours, compared with up to 250,000 iterations and about 8 hours for FixMatch with ACR.The reported outcome is a 2× speedup in total training time.
Parameter efficiency: LoFT updates only 0.18M trainable parameters out of 149.80M total parameters, whereas WideResNet-28-8 updates all 23.40M parameters.The comparison uses CLIP-ViT-B/16 as LoFT’s foundation model.