Source-linked AI summary
Federated Learning with Only Positive Labels
Felix X. Yu, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar
TL;DR
In federated multiclass learning, each user has positive data for only one class and lacks access to negative-class information. FedAwS adds server-side geometric regularization to Federated Averaging to separate class embeddings. On CIFAR-10 and CIFAR-100, it almost matches or comes very close to the performance of an oracle with access to all labels.
Problem
Federated multiclass learning is constrained when each user has data for only one class and cannot access other users’ class parameters.
Source: §1 · p. 1Method
FedAwS adds server-side geometric regularization to Federated Averaging to separate class embeddings, adapting the regularizer to focus on nearby classes for large output spaces.
Source: §4.1 · p. 5, §4.2 · p. 7Results
On CIFAR-10 and CIFAR-100, FedAwS almost matches or comes very close to the performance of the oracle method with access to all labels.
Source: §6.1 · p. 12Takeaways & Limitations
In extreme multiclass classification, the method replaces negative sampling with strong geometric regularization.
Source: §7 · p. 13FedAwS itself does not provide formal privacy guarantees, though federated differential privacy methods can be applied by adding noise to user updates.
Source: §2 · p. 3
Abstract
from arXiv · showhide
We consider learning a multi-class classification model in the federated setting, where each user has access to the positive data associated with only a single class. As a result, during each federated learning round, the users need to locally update the classifier without having access to the features and the model parameters for the negative classes. Thus, naively employing conventional decentralized learning such as the distributed SGD or Federated Averaging may lead to trivial or extremely poor classifiers. In particular, for the embedding based classifiers, all the class embeddings might collapse to a single point. To address this problem, we propose a generic framework for training with only positive labels, namely Federated Averaging with Spreadout (FedAwS), where the server imposes a geometric regularizer after each round to encourage classes to be spreadout in the embedding space. We show, both theoretically and empirically, that FedAwS can almost match the performance of conventional learning where users have access to negative labels. We further extend the proposed method to the settings with large output spaces.
1 Introduction
The paper studies federated multiclass learning when each user holds data for only one class and cannot access other users’ class parameters. It introduces FedAwS, which adds server-side geometric regularization to Federated Averaging to keep class embeddings separated.
1 Introduction
- Each user has only one class’s data and cannot access other users’ class embeddings, which can also be sensitive information.The server therefore cannot send each user the full classifier.Source: §1 · pp. 1–2
- Standard embedding-based classification needs positive instance–class pairs to score highly and negative pairs to score poorly.Users in this setting lack access to negative class embeddings.Source: §1 · p. 2
- Without negative-class information, vanilla federated learning can collapse all instances and classes to a single embedding point.The positive-only objective rewards small distances between instances and their positive classes without discouraging other classes from collapsing together.Source: §1 · p. 2
- FedAwS adds server-side geometric regularization after Federated Averaging to keep class embeddings separated.The paper also proposes modifying the regularizer for settings with many classes.Source: §1 · p. 2
2 Related Works
The paper distinguishes its positive-only federated multiclass setting from positive-unlabeled learning and one-class classification, and situates FedAwS alongside prior work on generative alternatives, spreadout regularization, and privacy.
2 Related Works
- Unlike positive-unlabeled learning, clients here lack unlabeled data spanning positive and negative classes; unlike one-class classification, the goal is collaborative multiclass learning.The related work cites Liu et al. (2002), Elkan & Noto (2008), Plessis et al. (2015), Hsieh et al. (2015), Moya & Hush (1996), and Manevitz & Yousef (2001).Source: §2 · p. 3
- The paper restricts attention to discriminative embedding-based classifiers, not generative approaches that require separate generation or synthesis of negatives.The authors describe those alternatives as data- and computation-consuming or as a separate, expensive federated GAN process.Source: §2 · p. 3
- FedAwS does not itself provide formal privacy guarantees, though the authors say federated differential privacy methods can be applied by adding noise to user updates.The paper cites Agarwal et al. (2018) and Abadi et al. (2016).Source: §2 · p. 3
- FedAwS combines stochastic negative mining, introduced by Reddi et al. (2019), with spreadout regularization used by Zhang et al. (2017) and Guo et al. (2019).The paper relates class separation to error-correcting output codes and contrasts its true-error result with Yu et al. (2013).Source: §2 · p. 3
3 Problem Setup
The setup defines the conventional federated classification workflow and scorer, then specifies how restricting users to one positive class changes available data and makes the usual objective degenerate.
3.1 Federated learning of a classification model
- The scorer maps an instance into an embedding space and uses class embedding vectors to produce scores for the classes.The scorer assigns a relevance score to each class; the class embedding is the corresponding row of the classification matrix.Source: §3.1 · pp. 3–4
- Each client holds local instance-label pairs drawn from its distribution, and the joint objective is to find a scorer that captures true class relevance.The local client minimizes an empirical estimate of risk based on its observations.Source: §3.1 · p. 4
- In conventional federated learning, a coordinator sends model parameters to clients, collects their updates, and aggregates them with Federated Averaging.The exposition restricts client weights to uniform weights.Source: §3.1 · p. 4
- Standard classification losses encourage instance embeddings to be close to positive class embeddings and far from negative class embeddings.The positive-only setting cannot use the latter objective because clients do not have negative class data or embeddings.Source: §3.1 · p. 4
3.2 Federated Learning with only positive labels
- For the positive-only setup, a client has its same-class examples, its own class embedding, and the current instance-embedding model parameters.It cannot access other clients’ label embeddings or share its data with them.Source: §3.2 · p. 5
- With only one class per client and no access to other clients’ label embeddings, positive-only training can drive all users and classes to identical embeddings.The analysis assumes one client per class for notation but also applies when multiple clients share a class.Source: §3.2 · p. 5
4 Algorithm
FedAwS adds server-side geometric regularization to Federated Averaging to prevent class embeddings from collapsing. For large output spaces, stochastic negative mining adaptively focuses the regularizer on nearby classes.
4.1 Federated Averaging with Spreadout (FedAwS)
- FedAwS prevents class embeddings from collapsing by adding a server-side geometric regularization step to Federated Averaging.The server separates different class embeddings by at least margin ν.Source: §4 · p. 5, §4.1 · p. 5
- Clients train with the positive part of the loss, while the server updates class embeddings and applies spreadout regularization.The spreadout regularizer together with the positive loss part accounts for instance similarity to positive and negative labels.Source: §4.1 · p. 6
4.2 FedAwS with stochastic negative mining
- For large class counts, full spreadout regularization is computationally expensive, and choosing its margin ν is problem-dependent.Source: §4.2 · p. 6
- The modified regularizer considers a subset of classes and, for each class, its k nearest embeddings, adapting the margin to the (k + 1)-th nearest class.This focuses separation on nearby classes.Source: §4.2 · p. 7
- Stochastic negative mining selects confusing classes from class embeddings and applies the regularizer at the server; the authors report it is crucial to FedAwS quality.Source: §4.2 · p. 7
5 Analysis
The analysis relates classification error to embedding geometry, establishes a classification-consistent cosine contrastive loss, and connects FedAwS to empirical risk minimization under stated assumptions.
5 Analysis
- Together, these results justify why FedAwS can approach the performance of a consistent classifier despite training only with positive labels.Source: §5 · p. 7
- Misclassification probability is bounded in terms of the expected distance from an instance embedding to its true class embedding and the minimum distance between class embeddings.The analysis identifies small instance-to-class distance and large inter-class separation as conditions implying low classification error.Source: §5 · p. 7
- Under unit-norm embeddings, setting α=1 and β=1 specializes the contrastive objective to cosine contrastive loss.The analysis relates this loss to classification consistency.Source: §5 · pp. 7–8
- The cosine contrastive loss is a valid surrogate for misclassification error, and similar analysis can establish its statistical consistency.The loss uses normalized embeddings; the analysis also extends to unnormalized embeddings, while normalization slightly improves empirical performance.Source: §5 · pp. 8–9
- With equal class counts and λ=1, FedAwS’s objective equals empirical risk for a loss whose negative-label contribution is independent of the input embedding.The empirical-risk equivalence is stated under the equal-count and λ=1 assumptions.Source: §5 · pp. 9–10
- The analysis can also extend to a contrastive loss that uses the k hardest labels as negatives rather than all non-y labels.Source: §5 · p. 11
6 Experiments
The experiments assess FedAwS on CIFAR and extreme-multiclass datasets, comparing positive-only baselines with oracle methods. FedAwS approaches oracle performance in both settings.
6 Experiments
- For FedAwS, the client-side squared hinge loss encourages positive instance-label pairs to have dot products above 0.9.The experiments use normalized class and instance embeddings.Source: §6 · p. 11
- Experiments compare FedAwS with positive-only squared-hinge baselines and Softmax, an oracle trained with both positive and negative labels.FedAwS uses stochastic negative mining; the fixed-embedding baseline prevents class-embedding collapse by holding embeddings at their random initialization.Source: §6 · p. 11
6.1 Experiments on CIFAR
- On CIFAR-10 and CIFAR-100, FedAwS nearly matches or comes close to the all-label oracle, while positive-only training collapses to a degenerate solution.On CIFAR-10 with ResNet-32, fixed random class embeddings perform almost identically to FedAwS and the oracle.Source: §6.1 · p. 12
6.2 Experiments on extreme-multiclass classification
- The extreme-multiclass evaluation converts multilabel datasets into multiclass tasks by uniformly sampling positive labels.The datasets have large numbers of classes and represent applications of federated learning with only positive labels.Source: §6.2 · p. 12
- FedAwS largely outperforms positive-only baselines and matches or nearly matches Softmax and SLEEC on AmazonCat and Amazon670K.On Amazon670K and WikiLSHTC, fixed random class embeddings do not work, making it crucial to train class embeddings with the rest of the model.Source: Table 3 · p. 12, §6.2 · p. 13
- Very large k worsens performance, small k may select positive labels and over-spread classes, and λ values of 10 or 100 sufficiently spread embeddings.These parameter effects were analyzed on AmazonCat.Source: §6.2 · p. 13, Table 4 · p. 13
7 Conclusion
The paper concludes that FedAwS learns high-quality classification models without negative instance-label pairs by geometrically spreading class embeddings on the server. Its regularization can replace negative sampling in extreme multiclass classification, and future work could use class-level features to generate embeddings.
7 Conclusion
- FedAwS learns high-quality classification models without negative instance-label pairs by geometrically spreading class embeddings on the server.The authors report theoretical and empirical justification for the method.Source: §7 · p. 13
- In conventional extreme multiclass classification, the proposed geometric regularization replaces negative sampling.The authors note that negative sampling techniques are crucial for conventional methods in this setting.Source: §7 · p. 13
- A future direction is extending id-based class embeddings to settings where class embeddings are generated from class-level features.Source: §7 · p. 13