PaperlayerView exact revision on arXiv

Source-linked AI summary

Federated Learning on Heterogeneous and Long-Tailed Data via Classifier Re-Training with Federated Features

Xinyi Shang, Yang Lu, Gang Huang, Hanzi Wang

arXiv:2204.13399v1cs.LG

TL;DR

Heterogeneous client data combined with long-tailed class distributions challenges federated learning, while class-distribution information used by many remedies raises privacy concerns. CReFF re-trains the classifier using balanced federated features rather than real data. The paper reports comparable performance to real-data retraining and improved results over state-of-the-art FL methods in this setting.

  • Problem

    Federated learning must contend with heterogeneous client data and long-tailed class distributions, while many long-tail methods rely on class-distribution information that may expose privacy-sensitive data.

    Source: §1 · p. 1

  • Method

    CReFF learns balanced federated features and re-trains the server-side classifier using them, without requiring real data or class-distribution information.

    Source: §1 · p. 2

  • Results

    CReFF outperforms state-of-the-art FL methods on heterogeneous, long-tailed image-classification tasks, and federated-feature retraining is comparable to real-data retraining.

    Source: §1 · p. 2, §5 · p. 6

  • Takeaways & Limitations

    The paper identifies the biased classifier as the primary factor behind poor global-model performance on heterogeneous and long-tailed data.

    Source: §1 · p. 2

    Using external balanced data on the server is impractical in real FL applications because transmitting client data violates privacy, and server-collected data may not match the client training distribution.

    Source: §3.2 · p. 3

Abstract

from arXiv · show

Federated learning (FL) provides a privacy-preserving solution for distributed machine learning tasks. One challenging problem that severely damages the performance of FL models is the co-occurrence of data heterogeneity and long-tail distribution, which frequently appears in real FL applications. In this paper, we reveal an intriguing fact that the biased classifier is the primary factor leading to the poor performance of the global model. Motivated by the above finding, we propose a novel and privacy-preserving FL method for heterogeneous and long-tailed data via Classifier Re-training with Federated Features (CReFF). The classifier re-trained on federated features can produce comparable performance as the one re-trained on real data in a privacy-preserving manner without information leakage of local data or class distribution. Experiments on several benchmark datasets show that the proposed CReFF is an effective solution to obtain a promising FL model under heterogeneous and long-tailed data. Comparative results with the state-of-the-art FL methods also validate the superiority of CReFF. Our code is available at https://github.com/shangxinyi/CReFF-FL.

1 Introduction

Heterogeneous client data and long-tailed class distributions make global-model training challenging, while many remedies rely on class-distribution information that raises privacy concerns. CReFF re-trains the classifier using balanced federated features, with the paper reporting results comparable to real-data retraining and better than state-of-the-art FL methods.

1 Introduction

  • Many existing long-tail methods require local or global class-distribution information, which may expose privacy-sensitive information.A shared balanced dataset on the server is also described as infeasible in most practical FL settings because of privacy concerns.Source: §1 · pp. 1–2
  • 18% was the highest performance gain from classifier re-training, at IF = 100 and α = 0.1, an extremely heterogeneous and long-tailed setting.The experiment re-trained the classifier using a small balanced dataset collected from clients; Figure 1 compares real data and synthesized federated features.Source: Figure 1 · p. 1, §1 · p. 2
  • CReFF optimizes balanced federated features to make their gradients close to real-data gradients, then uses those features to re-train the classifier.The paper reports comparable performance to retraining on real data and outperformance of state-of-the-art FL methods on heterogeneous, long-tailed image-classification tasks.Source: §1 · p. 2
  • The paper identifies the biased classifier as the primary factor behind poor global-model performance when data are both heterogeneous and long-tailed.Clients may hold different tail classes, and the server does not know local or global class distributions.Source: §1 · pp. 1–2

2 Related Work

Prior work addresses FL data heterogeneity through client/global optimization or server-side mechanisms, while long-tail methods often rebalance features or classifiers. The paper notes that many approaches depend on class distributions or auxiliary data unavailable or privacy-sensitive in FL.

2.1 Federated Learning with Heterogeneous Data

  • For heterogeneous FL data, prior work limits client/global model diversity or uses server-side mechanisms; these methods can perform poorly on tail classes without considering the global long-tail distribution.CCVR uses virtual features sampled from an approximated Gaussian mixture model.Source: §2.1 · p. 2
  • Some imbalanced-data FL methods require clients to upload class distributions, while Ratio loss uses balanced server auxiliary data; both requirements are problematic in practical FL.CReFF is contrasted as requiring neither real data nor class-distribution information.Source: §2.1 · p. 2

2.2 Long-tail learning

  • Long-tail methods rebalance features, re-weight classes, or retrain classifiers, but most require global class-distribution information that is infeasible to gather in FL.The cited approaches include feature augmentation, frequency re-weighting, and classifier retraining.Source: §2.2 · p. 2

3 Proposed Method

CReFF addresses heterogeneous, long-tailed federated learning by focusing on classifier bias and retraining the classifier on balanced federated features optimized to match real-feature gradients.

3.1 Preliminaries

  • The setup considers heterogeneous client partitions of long-tailed data and uses FedAvg to update the global model by weighted averaging selected client models.Source: §3.1 · pp. 2–3

3.2 Motivation

  • Around 15.35% was the performance gain from retraining the classifier, the largest gain among the five ResNet-8 components tested on CIFAR-10-LT.The experiment used 100 balanced examples per class and a FedAvg model trained with IF = 100 and α = 0.5.Source: §3.2 · p. 3, Figure 2 · p. 3
  • Using external data on the server is considered impractical; client data transmission violates FL privacy, while server-collected data may differ in distribution.Source: §3.2 · p. 3

3.3 Framework of the proposed CReFF

  • CReFF learns balanced federated features by matching their gradients to real-feature gradients, then retrains the classifier on those features.The goal is for the retrained classifier to perform comparably to one retrained on real features.Source: §3.3 · pp. 3–4
  • Each round, clients send updated models and real feature gradients; the server aggregates models, optimizes federated features, and returns updated global and retrained models.CReFF uses averaged class gradients, and does not require real data or local class-distribution information for classifier retraining.Source: §3.3 · pp. 3–4

4 Experimental Results

Experiments compare CReFF with federated-learning methods on long-tailed, heterogeneous benchmarks and examine why federated features work. CReFF leads the reported comparisons, while validation finds that its features match real-feature gradients without becoming similar to the real features.

4.1 Experimental Setup

  • Experiments use CIFAR-10/100-LT and ImageNet-LT, with Dirichlet partitions for CIFAR and different base models across datasets.CIFAR uses IF values 100, 50, and 10 with α = 0.5; ImageNet-LT uses α = 0.1. ResNet-8 is used for CIFAR and ResNet-50 for ImageNet-LT.Source: §4.1 · p. 5

4.2 Comparison with the State-of-the-art Methods

  • CReFF achieves the highest test accuracy on CIFAR-10-LT and CIFAR-100-LT across the reported imbalance factors, and the best results in all ImageNet-LT cases.At IF = 100, it improves over FedAvg by around 14.4% on CIFAR-10-LT and 4.3% on CIFAR-100-LT.Source: §4.2 · p. 5, Table 1 · p. 6, Table 2 · p. 6

4.3 Model Validation

  • Federated-feature and real-feature gradient dissimilarities decrease to around zero, while their feature cosine dissimilarity remains far from zero.This supports a similar classifier-retraining optimization path without requiring the federated features themselves to resemble real features.Source: §4.3 · pp. 5–6, Figure 4 · p. 6
  • More federated features generally improve accuracy, with gains stabilizing as their number increases; one feature per class still improves CIFAR-10-LT accuracy.With one feature per class, reported improvements are 4.21%, 3.74%, and 0.92% for IF = 100, 50, and 10, respectively.Source: §4.3 · p. 6, Figure 5 · p. 6

5 Conclusion

CReFF uses balanced federated features for privacy-preserving classifier re-training and is reported to outperform state-of-the-art FL methods on heterogeneous, long-tailed data.

5 Conclusion

  • CReFF learns a small set of balanced federated features on the server for privacy-preserving classifier re-training.The re-trained classifier can produce performance comparable to one re-trained on real data.Source: §5 · p. 6
  • The paper presents classifier re-training as an effective way to address data heterogeneity and long-tail distribution jointly in federated learning.Source: §5 · p. 6
  • Experiments report that CReFF outperforms state-of-the-art FL methods in the heterogeneous, long-tailed setting.Source: §5 · p. 6

Related papers

Loading 2204.13399v1…