PaperlayerView exact revision on arXiv

Source-linked AI summary

Federated Continual Learning with Weighted Inter-client Transfer

Jaehong Yoon, Wonyong Jeong, Giwoong Lee, Eunho Yang, Sung Ju Hwang

arXiv:2003.03196v5cs.LGstat.ML

TL;DR

Federated continual learning asks how clients learning private task streams can share useful knowledge without suffering interference from irrelevant tasks. FedWeIT separates shared and sparse task-specific parameters, then selectively transfers task knowledge across clients. The framework significantly outperforms the evaluated baselines while reducing communication cost.

  • Problem

    Federated continual learning studies clients that continuously learn private task sequences while needing inter-client knowledge transfer and protection against interference from irrelevant tasks.

  • Method

    FedWeIT decomposes model parameters into shared global parameters and sparse task-adaptive parameters, selectively combining task-specific knowledge received from other clients.

  • Results

    FedWeIT significantly outperforms single-machine continual-learning and naive federated continual-learning baselines across evaluated datasets and client counts, while using only 30% and 3% of dense-model parameters for bB and A, respectively.

  • Takeaways & Limitations

    Selective inter-client transfer enables positive use of relevant task knowledge while limiting interference, with sparse parameters making the framework communication-efficient.

  • Takeaways & Limitations

    The problem assumes an unknown task arrival order and access to each task only during its training period; naive federated aggregation can also suffer catastrophic forgetting.

Abstract

from arXiv · show

There has been a surge of interest in continual learning and federated learning, both of which are important in deep neural networks in real-world scenarios. Yet little research has been done regarding the scenario where each client learns on a sequence of tasks from a private local data stream. This problem of federated continual learning poses new challenges to continual learning, such as utilizing knowledge from other clients, while preventing interference from irrelevant knowledge. To resolve these issues, we propose a novel federated continual learning framework, Federated Weighted Inter-client Transfer (FedWeIT), which decomposes the network weights into global federated parameters and sparse task-specific parameters, and each client receives selective knowledge from other clients by taking a weighted combination of their task-specific parameters. FedWeIT minimizes interference between incompatible tasks, and also allows positive knowledge transfer across clients during learning. We validate our FedWeIT against existing federated learning and continual learning methods under varying degrees of task similarity across clients, and our model significantly outperforms them with a large reduction in the communication cost. Code is available at https://github.com/wyjeong/FedWeIT

1. Introduction

Federated continual learning extends continual learning to clients with private task streams, creating a need to transfer relevant knowledge while limiting interference from irrelevant client knowledge. FedWeIT addresses this by separating shared and sparse task-specific parameters and selectively combining knowledge across clients.

  • Problem: Federated continual learning lets multiple clients continuously learn from private task sequences while communicating task-specific parameters through a global server.This setting combines continual learning with federated parameter exchange rather than raw-data sharing.
  • Motivation: The framework is motivated by enabling continual learners to benefit from task knowledge acquired by other hospitals without compromising data privacy.A hospital learning a new disease task can transmit task-specific parameters for redistribution to other hospitals.
  • Problem: Inter-client transfer can help when knowledge is relevant but hinder training when clients share irrelevant knowledge.Relevant transferred knowledge can improve starting accuracy, convergence, and final accuracy, whereas irrelevant knowledge can cause interference.
  • Problem: Communication cost is a practical bottleneck in federated continual learning, motivating compact knowledge representations.Both client-to-server and server-to-client parameter transmission contribute to this cost.
  • FedWeIT: FedWeIT decomposes each local model into dense base parameters and sparse task-adaptive parameters.The base parameters encode task-generic knowledge, while sparse task-adaptive parameters encode task-specific knowledge.
  • FedWeIT: FedWeIT selectively combines task-adaptive parameters from other clients through weighted transfer to reduce interference and support relevant knowledge reuse.The framework also adaptively updates federated parameters and communicates sparse parameters for efficiency.

2. Related Work

Related work spans continual-learning methods for mitigating forgetting and federated-learning methods for distributed private-data training. Existing federated-learning research also emphasizes aggregation policies and communication-cost reduction.

  • Continual learning: Continual-learning methods address forgetting through regularization, architecture changes, or controlled optimization.Examples include EWC and IMM, network expansion methods such as DEN and RCL, and Stable SGD.
  • Federated learning: Federated learning trains a global model by aggregating parameters learned by clients on private data.FedAvg uses data-weighted averaging, while FedProx constrains local updates toward the global model.
  • Federated learning: Federated-learning research includes aggregation policies designed to reduce disparity across clients or improve aggregation quality.Examples include FedCurv and Bayesian non-parametric aggregation approaches.
  • Federated learning: Reducing communication cost is a crucial federated-learning challenge addressed by layer-wise aggregation and rapid-convergence algorithms.TWAFL aggregates shallow layers more frequently and deep layers near the end of a loop.

3. Federated Continual Learning with FedWeIT

Federated continual learning extends continual learning to multiple clients with private task streams, requiring knowledge transfer without inter-client interference. FedWeIT addresses this by decomposing parameters into global, client-base, and sparse task-adaptive components, while reducing communication through sparse transmission.

  • 3.1. Problem Definition: Federated continual learning trains multiple clients on private task sequences while communicating model parameters through a global server.The server aggregates client parameters and redistributes them to support inter-client knowledge transfer.
  • 3.2. Communicable Continual Learning: A single aggregated parameter can transfer irrelevant knowledge, causing inter-client interference and hindering continual learning across potentially unlimited task streams.Naive federated continual learning may also suffer catastrophic forgetting inherited from continual learning.
  • 3.3. Federated Weighted Inter-client Transfer: FedWeIT decomposes model parameters into global parameters, client-specific base parameters, and task-adaptive parameters for each client task.The base parameters capture generic knowledge, while task-adaptive parameters capture task-specific knowledge.
  • 3.3. Federated Weighted Inter-client Transfer: Sparse masks selectively extract relevant components from global knowledge, reducing interference while enabling efficient communication.The base parameter is initialized from the previous global parameter, then masked for the current task; task-adaptive parameters provide additional task-specific knowledge.
  • 3.4. Efficient Communication via Sparse Parameters: FedWeIT requires |C| × (R × |bB| + |A|) for client-to-server communication, versus |C|×R×|θ| for naive federated continual learning baselines.For server-to-client communication, FedWeIT requires |C| × (R × |θG| + (|C| −1) × |A|), with θG and A highly sparse.

4. Experiments

FedWeIT is evaluated across heterogeneous federated continual-learning settings, comparing adaptation, forgetting, transfer, efficiency, and communication against continual-learning and federated baselines. It consistently outperforms competing methods while using sparse parameter transfer and selective inter-client knowledge.

  • Task adaptation: FedWeIT consistently outperforms all baselines with 100 clients by selectively using other clients’ knowledge for rapid target-task adaptation.The reported improvement is attributed to selective utilization of task-specific knowledge from other clients.
  • Efficiency: FedWeIT achieves higher accuracy with fewer parameters than FedProx-APD by reusing task-adaptive parameters from other clients.This reuse is identified as the main source of the model’s parameter efficiency.
  • Communication cost: FedWeIT uses 30% of dense-model parameters for bB and 3% for A, making it more communication-efficient than federated continual-learning baselines.The communication costs include client-to-server and server-to-client transmission for training each task.
  • Catastrophic forgetting: FedWeIT shows no sign of catastrophic forgetting, whereas naive federated continual-learning baselines experience stronger forgetting from inter-client interference.FedProx-APD also avoids catastrophic forgetting but has inferior performance because of ineffective knowledge transfer.
  • Inter-client transfer: Attention weights assign larger values to task parameters from the same or similar datasets, indicating selective weighted inter-client transfer.The analysis examines attention at the first fully connected layer for MNIST and CIFAR-100 task splits.

5. Conclusion

The conclusion frames federated continual learning as continual local learning augmented by indirect experience from other clients. FedWeIT addresses interference and transfer through shared global parameters, sparse task-adaptive parameters, and selective parameter use.

  • 5. Conclusion: Federated continual learning lets local models use task knowledge from other clients while learning continuously on private task sequences.The setting introduces inter-client knowledge transfer alongside continual learning.
  • 5. Conclusion: FedWeIT decomposes each client’s model into shared global parameters and sparse task-adaptive parameters specific to each task.Clients selectively update shared parameters and use task-adaptive parameters from other clients.
  • 5. Conclusion: Across varied task similarities, FedWeIT significantly outperforms federated and continual-learning baselines with reduced communication cost.The conclusion presents this as the outcome of the experimental validation.

Supplementary File

The supplementary file expands the paper’s experimental documentation and reports additional results. It covers implementation details, datasets, metrics, communication frequency, and an Overlapped-CIFAR-100 ablation.

  • Supplementary File: The supplement is organized to provide explanations and experiments not covered in the main document.Its stated purpose is to extend the main paper’s descriptions and results.
  • Section A: Section A provides experimental details covering network architecture, training configurations, forgetting measures, and datasets.These details supplement the main document’s experimental description.
  • Section B: Section B reports additional experiments on communication frequency and an Overlapped-CIFAR-100 ablation study.These are listed as supplementary experimental results.

A. Experimental Details

The experiments use modified LeNet and ResNet-18 backbones across Overlapped-CIFAR-100 and NonIID-50 task configurations. Evaluation measures averaged accuracy and forgetting, alongside parameter and communication costs.

  • Network Architecture: The study validates FedWeIT with modified LeNet and conventional ResNet-18 backbone architectures.LeNet is used for both datasets, while ResNet-18 is additionally used with NonIID-50.
  • Configurations: Training uses Adam with adaptive learning-rate decay, task transitions at a learning-rate threshold, and fixed task-specific hyperparameters.The configurations specify decay every five epochs without validation-loss decrease and early transition when the learning rate reaches ρ.
  • Configurations: Experiments with 20 and 100 clients reduce minibatch size to 10 and use client fractions 0.25 and 0.05 per communication round.These settings differ from the five-client configuration.
  • Metrics: Averaged Accuracy is the mean test accuracy across tasks after learning task t, while Averaged Forgetting measures average disparity from minimum task accuracy during training.Both metrics follow continual-learning evaluation practice.
  • Datasets: Overlapped-CIFAR-100 uses 20 superclass-based non-IID tasks, while NonIID-50 combines eight heterogeneous datasets into 50 non-IID tasks.Tasks are distributed across clients after selecting or splitting task instances according to the experimental configuration.
  • Additional experimental details: Supplementary experiments include 20- and 100-client task-adaptation comparisons and a 20-task Overlapped-CIFAR-100 result table.The cited supplementary figures and table extend the main experimental settings.

B. Additional Experimental Results

The analysis examines how communication frequency affects FedWeIT performance and communication efficiency. More frequent communication improves performance but increases communication cost, creating a trade-off.

  • Communication Frequency: 1, 2, 5, and 20 training epochs per communication round are compared for FedWeIT on Overlapped-CIFAR-100.The experiments evaluate four communication-frequency settings.
  • Communication Frequency: More frequent parameter updates produce higher performance while maintaining smaller network capacity.Frequent communication efficiently updates parameters by transferring inter-client knowledge.
  • Communication Efficiency: 16.9 times larger entire communication cost is required for 1 epoch per round than for 20 epochs per round.The result illustrates the cost of choosing more frequent communication.
  • Baseline Comparison: FedWeIT variants consistently outperform federated continual learning baselines despite differing communication frequencies.The comparison concerns performance under the tested communication settings.

B.2. Ablation Study with Model Components

The component ablation study evaluates parameter decomposition and selective transmission. Removing either base or task-adaptive parameter communication lowers performance, while removing the sparse mask lowers accuracy and raises capacity and cost.

  • Study Design: The ablation study compares four model variations on the NonIID-50 dataset.The variants remove base communication, task-adaptive communication, sparse transmission, or the sparse vector mask.
  • Parameter Communication: Removing communication of either base parameter B or task-adaptive parameters produces significantly lower performance than the full model.Both components contribute to inter-client knowledge transfer.
  • Parameter Communication: The w/o A variant obtains very low performance because it suffers from catastrophic forgetting.This variant communicates only sparse local base parameters.
  • Selective Transmission: Removing the sparse vector mask m achieves lower accuracy with larger capacity and cost.The result demonstrates the importance of selective transmission.

B.3. Ablation Study with Regularization Terms

The regularization ablation shows that the ℓ1 and ℓ2 terms impose different trade-offs. Removing ℓ1 improves performance but increases memory, whereas removing ℓ2 causes forgetting.

  • Regularization Effects: Removing the ℓ1 term achieves better performance but requires significantly larger memory.The ablation is reported in Table B.8 on NonIID-50.
  • Regularization Effects: Removing the ℓ2 term causes the method to suffer from forgetting.The ℓ2 term therefore contributes to retaining previously learned knowledge.
  • Study Design: The regularization ablation evaluates the proposed terms through the analysis reported in Table B.8.The study isolates the effects of the ℓ1 and ℓ2 regularizers.

B.4. Forgetting Analysis

The forgetting analysis compares FedWeIT with local and federated continual-learning baselines as tasks accumulate. EWC-based methods degrade over subsequent tasks, while APD-based methods and FedWeIT prevent catastrophic forgetting across the tested task sequences.

  • Forgetting Comparison: EWC-based local, FedAvg, and FedProx methods show performance degradation while learning new tasks.The analysis considers performance on earlier tasks as the number of learned tasks increases.
  • Forgetting Prevention: APD-based methods and FedWeIT prevent catastrophic forgetting regardless of how many tasks or which tasks clients subsequently learn.The figure analyzes tasks 1 through 9 while excluding the last task.
Loading 2003.03196v5…