PaperlayerView exact revision on arXiv

Source-linked AI summary

FedBST: Bilateral Self-Training for Unsupervised Federated Classification

Aijing Yu, Kuangpu Guo, Jian Liang, Xiao-Yu Zhang

arXiv:2610.30634v1cs.CV

TL;DR

FedBST targets unsupervised federated classification under severe Non-IID shifts, where high-confidence-only pseudo-labeling leaves ambiguous samples unused. It combines uncertainty-aware confidence penalization with a visual-text ensemble, and reports consistent improvements over existing methods across multiple benchmarks.

  • Problem

    Under Non-IID shifts, high-confidence-only pseudo-labeling neglects informative ambiguous samples, while global textual prompts may not capture client-specific visual variation.

    Source: §I · p. 1

  • Method

    FedBST regularizes ambiguous predictions with uncertainty-aware confidence penalization and combines local visual prototypes with generic textual prototypes at inference.

    Source: §I · p. 1, B. Uncertainty-Aware Confidence… · p. 2, C. Visual-Text Ensemble · p. 3

  • Results

    FedBST consistently achieves the best performance across the reported Dirichlet-based label-skew settings and improves over existing methods across multiple benchmarks.

    Source: §I · p. 2, B. Experimental Results · p. 4

  • Takeaways & Limitations

    The results support using both high-confidence and ambiguous samples, together with visual-text ensembling, for robust unsupervised federated classification with CLIP.

    Source: §V · p. 6

Abstract

from arXiv · show

Unsupervised federated learning with Vision-Language Models (e.g., CLIP) offers a promising solution for label-free classification, yet its performance often degrades under severe non-IID shifts. A pioneering method, FedCoPL, typically relies on self-training strategies that exclusively prioritize high-confidence predictions for pseudo-labeling. However, this selective approach neglects the information of ambiguous samples (often referred to as ``waist" samples), leading to the underutilization of data and biased decision boundaries. To address this limitation, we propose FedBST, a bilateral self-training framework that effectively exploits both high-confidence and ambiguous data. Specifically, for local optimization, we introduce uncertainty-aware confidence penalization (UACP). UACP targets ambiguous samples by penalizing overconfident predictions to implicitly promote higher uncertainty, thereby preventing erroneous pseudo-labeling. At inference, we employ a visual-text ensemble strategy. This approach aggregates uncertainty-aware local visual prototypes and ensembles them with textual prototypes, achieving lightweight adaptation to local distributions via prototype-based calibration. Experiments demonstrate that FedBST achieves consistent improvements over existing methods with standard federated aggregation protocols.

I. INTRODUCTION

Label-free classification is valuable for federated learning, but existing approaches leave direct classification and ambiguous data underused. FedBST addresses this with bilateral self-training and visual-text inference, and reports improvements in accuracy and robustness.

I. INTRODUCTION

  • FedCoPL uses CLIP zero-shot predictions for cooperative pseudo-labeling but selects high-confidence predictions and discards uncertain ones.Conventional federated learning also depends on annotated data, which can be impractical where manual labeling is costly and privacy constraints are strict.Source: §I · p. 1
  • Under Non-IID shifts, excluding ambiguous samples can bias decision boundaries, while global textual prompts may miss client-specific visual variation.The introduction identifies both limitations as sources of reduced adaptability to local data.Source: §I · p. 1
  • FedBST uses both high-confidence and ambiguous data, unlike methods that rely solely on positive learning from high-confidence samples.The framework is designed to address underused ambiguous samples and improve inference reliability.Source: §I · pp. 1–2
  • FedBST combines uncertainty-aware confidence penalization for ambiguous samples with visual-text ensembling of local visual and generic textual prototypes.UACP penalizes overconfident predictions to promote higher predictive entropy; the ensemble adapts inference to local visual distributions without additional parameter optimization.Source: §I · pp. 1–2
  • FedBST outperforms state-of-the-art methods in accuracy and robustness across multiple benchmarks with standard server-side aggregation protocols.The introduction reports the overall comparison without giving numerical results.Source: §I · p. 2

II. RELATED WORK

Prior unsupervised federated learning methods largely target representation learning, while supervised vision-language adaptation and selective pseudo-labeling leave gaps for label-free classification. FedBST builds on FedCoPL by incorporating ambiguous samples alongside high-confidence data.

II. RELATED WORK

  • FedU and FedEMA address heterogeneity through divergence-aware updates, while other UFL methods use knowledge distillation, or clustering.These representation-focused approaches are not directly applicable to downstream classification without labels.Source: §II · p. 2
  • PromptFL and FedCLIP optimize prompts but require supervision, and personalization methods – also depend on ground truth.These requirements make the approaches unsuitable for fully label-free scenarios.Source: §II · p. 2
  • FedCoPL uses CLIP zero-shot predictions and global budget allocation for label-free classification, but its high-confidence selection neglects ambiguous samples.FedBST builds on this framework with bilateral self-training that exploits both high-confidence and ambiguous data.Source: §II · p. 2

A. Preliminaries

The preliminaries define the unlabeled, multi-client CLIP setting and its prompt-based prediction objective. They also describe CoPL’s budgeted selection of high-confidence samples and FedBST’s motivation to regularize the ambiguous samples left out.

A. Preliminaries

  • The setting has K clients with private unlabeled datasets, a shared pretrained CLIP backbone, and learnable visual and client-specific textual prompts.The image and text encoders are fixed, and the prompts are optimized collaboratively without ground-truth labels.Source: A. Preliminaries · p. 2
  • CoPL [11] uses global budget allocation to balance clients and selects the highest-confidence samples for cross-entropy optimization.This selection excludes samples outside the budget, including ambiguous samples near decision boundaries.Source: A. Preliminaries · p. 2
  • FedBST adds UACP to the previously ignored ambiguous samples, which can otherwise leave their predictions without explicit constraints.The preliminaries describe this as a way to address overly confident decision boundaries.Source: A. Preliminaries · p. 2

B. Uncertainty-Aware Confidence Penalization

FedBST identifies informative ambiguous samples among those excluded from positive pseudo-labeling, then uses them to regularize confidence without assigning class labels. This complements the high-confidence training loss while preserving client-specific textual prompts during aggregation.

B. Uncertainty-Aware Confidence Penalization

  • A dual filter selects ambiguous candidates from the remaining local data, separating informative waist samples from low-quality noise.The selection combines a symmetric confidence interval with an entropy threshold.Source: B. Uncertainty-Aware Confidence… · pp. 2–3
  • UACP minimizes the maximum predicted class probability for ambiguous samples, discouraging overconfident predictions without imposing class-specific supervision.The confidence penalty acts as a lightweight surrogate for reducing peaked predictive distributions.Source: B. Uncertainty-Aware Confidence… · p. 3
  • The local objective combines the base loss on high-confidence samples with UACP regularization on ambiguous samples, with λ balancing the two.This is the confidence-penalization component shown in the framework overview.Source: B. Uncertainty-Aware Confidence… · p. 3, Figure 1 · p. 3
  • After local updates, visual prompts are aggregated globally while textual prompts remain client-specific to preserve local semantic adaptation.Source: B. Uncertainty-Aware Confidence… · p. 3

C. Visual-Text Ensemble

FedBST augments global textual semantics with visual prototypes built from reliable local predictions. Its inference ensemble restricts the visual stream to locally observed classes while retaining textual coverage for the full label space.

C. Visual-Text Ensemble

  • After local training, each class prototype is formed by summing visual features of high-confidence samples predicted as that class.The prototype is then ℓ2-normalized, and a class-existence mask marks whether the client has a prototype for that class.Source: C. Visual-Text Ensemble · p. 3
  • The textual stream supplies global category semantics across the full label space, including classes absent from a client's local data.Source: C. Visual-Text Ensemble · pp. 3–4
  • The visual stream normalizes probabilities only over locally observed classes, assigning zero visual probability to missing classes.The class-existence mask restricts the visual probability distribution to classes present on the client.Source: C. Visual-Text Ensemble · pp. 3–4
  • Final inference linearly combines the textual and visual streams, refining predictions for observed classes with local prototypes while retaining textual knowledge for missing classes.Source: C. Visual-Text Ensemble · p. 4

A. Experimental Setup

The experiments evaluate unsupervised federated classification across six benchmarks under two non-IID label-skew scenarios, comparing FedBST with federated and pseudo-labeling baselines.

A. Experimental Setup

  • Evaluation covers DTD, RESISC45, UCF101, CUB, CIFAR-10, and CIFAR-100 benchmarks.Source: A. Experimental Setup · p. 4
  • The study simulates Dirichlet-based skew and quantity-based skew to represent non-IID client data distributions.Dirichlet skew varies β; quantity skew distributes class-grouped samples through heterogeneous shards.Source: A. Experimental Setup · p. 4
  • Comparisons include FedCoPL, FedOPT, pFedPrompt, PromptProx, and PromptFL, with supervised baselines adapted using FPL or CPL pseudo-labeling.Source: A. Experimental Setup · p. 4
  • Experiments use frozen CLIP ViT-B/32 by default, optimize prompts with SGD, and report mean accuracy over three trials.Default values are τcon = 0.2, τent = 0.5, λ = 0.001, and α = 0.8.Source: A. Experimental Setup · p. 4

B. Experimental Results

FedBST performs best across the reported label-skew settings, with a clear margin over FedCoPL under quantity-based class scarcity. Its advantage is also reported under Dirichlet skew, including when high-confidence data is scarce.

Results under Dirichlet-based label skew with various datasets

  • FedBST achieves the best results under Dirichlet-based label skew and outperforms the strong FedCoPL baseline consistently.The reported settings use β ∈ {0.1, 0.05}.Source: B. Experimental Results · p. 4

Results under quantity-based label skew with various datasets

  • FedBST outperforms FedCoPL by an average of 0.95% under quantity-based skew, where clients observe only a limited subset of classes.Several baselines underperform zero-shot CLIP in this class-scarce setting.Source: B. Experimental Results · p. 4

Results under different client numbers and joining rates

  • The results cover experiments varying client numbers and client joining rates under Dirichlet-based label skew with β = 0.1.Source: Figure 2 · p. 5

C. Analysis

FedBST maintains the highest accuracy as client numbers increase and outperforms competitors when participation is sparse. The analysis also reports stable performance across hyperparameter settings, gains from both components, and results across image-encoder architectures.

Results under different client numbers

  • As client numbers increase, FedBST maintains the highest accuracy and degrades more slowly than other methods.The passage attributes this scalability to UACP preventing noise accumulation as local data becomes sparse.Source: C. Analysis · p. 5

Impact of client joining rates

  • FedBST consistently outperforms competitors under varying client participation rates, with VTE reported to ensure stability when active clients are sparse.Source: C. Analysis · p. 5

Results under different image encoder backbones

  • FedBST outperforms zero-shot CLIP and federated baselines with both ViT-B/16 and RN50 image encoders.The results are reported under Dirichlet-based label skews.Source: C. Analysis · p. 5

Parameter Sensitivity

Effectiveness of each component

  • UACP and VTE each provide gains, and their combination performs best under Dirichlet-based label skew with β = 0.1.UACP regularizes training, while VTE calibrates inference.Source: C. Analysis · p. 6, Table VII · p. 6

V. CONCLUSION

FedBST uses both high-confidence and uncertain samples for unsupervised federated classification under severe non-IID conditions. Across multiple benchmarks, it delivers stable improvements, with the largest gains under strong label skew and limited data.

  • FedBST combines high-confidence learning with uncertainty-based regularization and visual-text ensembling, with stable improvements across benchmarks and largest gains under strong label skew and limited data.Source: §V · p. 6

Related papers

Loading 2610.30634v1…