PaperlayerView exact revision on arXiv

Source-linked AI summary

Federated Class-Incremental Learning

Jiahua Dong, Lixu Wang, Zhen Fang, Gan Sun, Shichao Xu, Xiao Wang, Qi Zhu

arXiv:2203.11473v1cs.LG

TL;DR

Federated class-incremental learning must handle continuously arriving classes, limited client memory, and unseen classes from new clients without undermining privacy. GLFC combines local forgetting compensation, semantic distillation, proxy-server selection, and prototype-based communication, achieving 4.4%∼15.1% higher average accuracy on benchmark datasets.

  • Problem

    Federated class-incremental learning lacks methods suited to continuously arriving classes, limited old-class storage, and newly participating clients with unseen classes.

  • Method

    GLFC combines class-aware gradient compensation and semantic relation distillation with proxy-server selection and prototype gradient-based privacy-preserving communication.

  • Results

    4.4%∼15.1% improvement in average accuracy is reported on several benchmark datasets.

  • Takeaways & Limitations

    GLFC is designed to preserve old-class knowledge while learning new classes in federated settings with local and cross-client imbalance.

  • Takeaways & Limitations

    Attackers may reconstruct raw prototype samples from stolen gradients used in prototype-based communication.

Abstract

from arXiv · show

Federated learning (FL) has attracted growing attention via data-private collaborative training on decentralized clients. However, most existing methods unrealistically assume object classes of the overall framework are fixed over time. It makes the global model suffer from significant catastrophic forgetting on old classes in real-world scenarios, where local clients often collect new classes continuously and have very limited storage memory to store old classes. Moreover, new clients with unseen new classes may participate in the FL training, further aggravating the catastrophic forgetting of the global model. To address these challenges, we develop a novel Global-Local Forgetting Compensation (GLFC) model, to learn a global class incremental model for alleviating the catastrophic forgetting from both local and global perspectives. Specifically, to address local forgetting caused by class imbalance at the local clients, we design a class-aware gradient compensation loss and a class-semantic relation distillation loss to balance the forgetting of old classes and distill consistent inter-class relations across tasks. To tackle the global forgetting brought by the non-i.i.d class imbalance across clients, we propose a proxy server that selects the best old global model to assist the local relation distillation. Moreover, a prototype gradient-based communication mechanism is developed to protect privacy. Our model outperforms state-of-the-art methods by 4.4%-15.1% in terms of average accuracy on representative benchmark datasets.

1. Introduction

Federated class-incremental learning addresses dynamic, privacy-preserving federated training when clients receive new classes, have limited memory, and may join with unseen classes. GLFC compensates forgetting locally and globally through class-aware losses, relation distillation, a proxy server, and privacy-preserving prototype communication.

  • Existing federated learning methods generally assume that the overall class set is fixed over time.
  • Dynamic new-class arrivals make storing all old-class data costly, while limited client memory can cause catastrophic forgetting.
  • New clients with unseen classes and non-i.i.d. class distributions across clients further complicate global class-incremental learning.
  • Simply combining federated learning and class-incremental learning can expose privacy-sensitive information about when and where new classes arrive.
  • GLFC addresses local and global forgetting using class-aware gradient compensation, semantic relation distillation, proxy-server model selection, and prototype gradient-based communication.
  • 4.4%∼15.1% improvement in average accuracy is reported on several benchmark datasets.

2. Related Work

Related work covers federated learning and class-incremental learning, but existing approaches do not directly address federated class-incremental learning under decentralized, dynamically arriving classes.

  • Federated learning trains a global model by aggregating local model parameters across decentralized clients.
  • Existing federated learning methods cannot effectively learn new classes continuously when local clients have limited memory for old classes.
  • Class-incremental learning learns new classes continuously while addressing forgetting on old classes.
  • Class-incremental methods cannot directly solve FCIL because they assume knowledge of when and where new-class data arrive.

3. Problem Definition

FCIL models sequential, non-i.i.d. class arrivals across existing and newly added clients that collaboratively train a private global model without prior knowledge of task timing or client additions.

  • FCIL has streaming tasks whose current label space contains new classes distinct from classes learned in previous tasks.
  • Each client may retain an exemplar memory containing selected exemplars for old classes.
  • Selected clients receive the latest global model, train locally on private data, upload updated models, and receive subsequent aggregated parameters.
  • Client class distributions are non-i.i.d., and local client label spaces are subsets of the global task label space.
  • Clients are divided into old-memory clients, clients with current new data and old exemplars, and newly added clients without old-class memory.
  • The number of streaming tasks, class distributions, new-class arrival times, and client additions are unknown in advance.
  • The FCIL goal is to learn new classes consecutively while alleviating old-class forgetting under privacy-preserving communication with the central server.

4. The Proposed GLFC Model

GLFC compensates catastrophic forgetting in federated class-incremental learning from both local and global perspectives. It balances class-specific local updates, preserves inter-class relations across tasks, and uses a proxy server to select an old global model while communicating through perturbed prototypes.

  • Model Overview: GLFC addresses local forgetting with class-aware gradient compensation and class-semantic relation distillation, while a proxy server addresses global forgetting.The model targets class imbalance at local clients and non-i.i.d. class imbalance across clients.
  • Class-Aware Gradient Compensation: Class imbalance causes different learning and forgetting rates across classes, worsening local forgetting as new streaming data becomes old.The method treats local forgetting as imbalanced gradient propagation in the output layer.
  • Class-Aware Gradient Compensation: LGC normalizes learning rates for new classes and forgetting rates for old classes by reweighting their gradient propagation.Separate gradient normalization is applied to old and new classes before reweighting the cross-entropy loss.
  • Class-Semantic Relation Distillation: LRD preserves semantic consistency by distilling inter-class relations between old and new classes across incremental tasks.Unlike distillation methods that focus only on old-class consistency, LRD considers old-new relations using smoothed labels.
  • Task Transition Detection: Task transition detection identifies newly arriving classes using a sudden rise in average prediction entropy, then updates task state, exemplar memory, and the stored old model.The method empirically sets the entropy threshold r_h to 1.2.
  • Global Catastrophic Forgetting Compensation: Non-i.i.d. class imbalance creates heterogeneous forgetting across clients, so the proxy server selects the best old global model for relation distillation.Clients send perturbed new-class prototypes through prototype gradient-based communication; the proxy server reconstructs them and monitors global-model performance.
  • Private Communication: Prototype perturbations protect privacy because attackers obtain little useful information even if they reconstruct the perturbed prototypes.The privacy mechanism addresses the risk of reconstructing raw prototype samples from shared gradients.

5. Experiments

Experiments evaluate GLFC against class-incremental baselines across three datasets, task counts, ablations, and exemplar-memory sizes. GLFC consistently improves average accuracy and benefits from larger exemplar memories.

  • Performance Comparison: GLFC outperforms existing class-incremental methods by 4.4%∼15.1% in average accuracy across the reported FCIL comparisons.The comparisons use CIFAR-100, ImageNet-Subset, and TinyImageNet under matched protocols and a ResNet-18 backbone.
  • Ablation Studies: Removing any GLFC module reduces performance by 1.1%∼10.1%, supporting the contribution of LGC, LRD, and the proxy server.The ablations identify all modules as essential to training a global class-incremental model.
  • Qualitative Analysis of Incremental Tasks: GLFC performs better than competing baselines for T = 5, T = 10, and T = 20 incremental tasks in the qualitative analyses.The analyses cover CIFAR-100 and ImageNet-Subset.
  • Exemplar Memory Analysis: Increasing CIFAR-100 exemplar memory from 500 to 2,000 improves GLFC’s performance on streaming new-class learning.The results associate larger local exemplar memories with stronger memory for old classes.
  • Exemplar Memory Analysis: GLFC’s task-transition detection and exemplar-memory updates support identifying new classes during streaming learning.This finding is reported alongside the exemplar-memory study.

6. Conclusion

The paper formulates FCIL and proposes GLFC to address local and global catastrophic forgetting. Its experiments on representative benchmark datasets demonstrate the model’s effectiveness.

  • Conclusion: GLFC addresses local and global catastrophic forgetting in Federated Class-Incremental Learning.The model combines local compensation and global model selection mechanisms.
  • Conclusion: Class-aware gradient compensation and class-semantic relation distillation correct imbalanced gradients and preserve inter-class relations across tasks.A proxy server selects the best old model to preserve old-class memory.
  • Conclusion: Experiments on representative benchmark datasets demonstrate the effectiveness of GLFC.

A. Implementation Details

The experiments use CIFAR-100, ImageNet-Subset, and TinyImageNet as benchmark datasets for evaluating FCIL methods.

  • Datasets: CIFAR-100 contains 60,000 color images from 100 classes, with 500 training and 100 evaluation samples per class.Images have size 32 × 32.
  • Datasets: ImageNet-Subset contains 100 sampled classes, with 500 training and 100 test samples per class.Images have size 224 × 224.
  • Datasets: TinyImageNet contains 100,000 samples across 200 classes, with 500 training and 50 test samples per class.Each sample is downsized to 64 × 64.

A.2. Experimental Settings

The experimental settings define federated client participation, incremental-task protocols, comparison methods, and the optimization pipeline used to evaluate GLFC.

  • Federated Settings: Each global round randomly selects local clients, which perform local training before sharing updated models for global aggregation.The first task uses 30 clients and selects 10 clients for 20 local epochs per round.
  • Comparison Protocol: The study follows established class-incremental protocols, identical iCaRL class order, and ResNet-18 for fair comparisons.The comparison uses top-1 accuracy and includes six state-of-the-art class-incremental methods.
  • Optimization Pipeline: The central server sends the current global model to selected clients, while a proxy server and gradient encoding model support the training pipeline.The detailed optimization pipeline is specified in Algorithm 1.
  • Optimization Pipeline: When clients detect new classes, they update exemplar memories and use the current global model; otherwise, they continue local optimization with the prior model.The resulting local models are used to obtain the next global model.
  • TinyImageNet Results: On TinyImageNet, GLFC exceeds other state-of-the-art methods by 4.7%∼11.0% in average accuracy.The reported improvement covers the evaluated incremental tasks.

C.2. Ablation Studies

Ablation results on TinyImageNet show that each GLFC component contributes to mitigating local or global catastrophic forgetting, with the proxy server addressing cross-client forgetting.

  • Removing class-aware gradient compensation lowers average accuracy by 2.7%∼2.8%, validating its role in correcting imbalanced gradient propagation.The loss compensates for forgetting caused by class imbalance at local clients.
  • Removing class-semantic relation distillation reduces average accuracy by 10.1%∼10.2%, confirming its importance for inter-class semantic consistency across tasks.This module addresses local catastrophic forgetting by distilling consistent relations across incremental tasks.
  • Removing the proxy server decreases average accuracy by 3.2%∼4.6%, supporting its role in addressing global forgetting from non-i.i.d. class imbalance.The proxy server selects an old global model to assist local relation distillation.
  • All three proposed components cooperate effectively, since removing any one causes significant performance degradation.The ablation compares Ours with Ours-w/oCGC, Ours-w/oCRD, and Ours-w/oPRS on TinyImageNet.

C.3. Effects of Incremental Tasks

Across TinyImageNet experiments with different numbers of incremental tasks, GLFC maintains the best performance and improves average accuracy over other state-of-the-art methods.

  • 3.2%∼10.0% average-accuracy improvement over other state-of-the-art methods is observed across all TinyImageNet incremental tasks.The comparison covers settings with T = 5 and T = 10.
  • GLFC achieves the best performance under both T = 5 and T = 10 incremental-task settings.These results are reported in Tables 7 and 8.
  • The results indicate that GLFC addresses both local and global catastrophic forgetting in the FCIL setting.The reported improvement remains consistent across different numbers of incremental tasks.

D. Qualitative Analysis of Exemplar Memory

On CIFAR-100 with T = 10, larger local exemplar memories improve GLFC’s performance across incremental tasks, while task-transition detection updates the memory for new classes.

  • GLFC performs better across all incremental tasks when local clients have larger exemplar-memory storage.The experiment evaluates Ml values of 500, 1000, 1500, and 2000.
  • Task-transition detection enables GLFC to distinguish new classes and update the corresponding local exemplar memory.The updated memory supports retention of old-class knowledge.
  • The updated local exemplar memory plays an essential role in tackling local catastrophic forgetting on old classes.

E.1. Limitation

The paper presents GLFC algorithmically but does not yet provide mathematical theoretical support for the FCIL problem or the proposed model.

  • The paper’s main limitation is its algorithmic focus, without mathematical theoretical support for FCIL or GLFC.
  • Developing theory is difficult because rare theoretical analysis exists for conventional class-incremental learning.The authors propose pursuing FCIL analysis from a federated-learning perspective in future work.
Loading 2203.11473v1…