Source-linked AI summary
Federated Continual Learning via Knowledge Fusion: A Survey
Xin Yang, Hao Yu, Xin Gao, Hao Wang, Junbo Zhang, Tianrui Li
TL;DR
Federated learning and continual learning address, respectively, collaborative model training across data silos and learning from changing task streams, but their integration must contend with forgetting across tasks and clients. This survey frames FCL as knowledge fusion, organizes methods into synchronous and asynchronous frameworks and seven fusion categories, and uses FedAvg experiments to examine spatial-temporal forgetting. Its analysis distinguishes client-distribution differences associated with spatial forgetting from task-distribution differences associated with temporal forgetting.
Problem
Method
The survey examines FCL through FedAvg experiments and organizes existing methods into synchronous and asynchronous frameworks and knowledge-fusion categories.
Source: §1 · p. 2, §6 · p. 15Results
The analysis associates client-distribution divergence with spatial forgetting and task-distribution differences with temporal forgetting.
Source: §3.2.4 · p. 6Takeaways & Limitations
The paper concludes that greater client-data variance makes effective knowledge fusion more important than straightforward aggregation for retaining seen-class knowledge.
Source: §3.2.4 · p. 6The survey notes that parameter-only weighted averaging can expose sensitive information and does not facilitate knowledge integration between local and global models.
Source: §5.3 · p. 10
Abstract
from arXiv · showhide
Data privacy and silos are nontrivial and greatly challenging in many real-world applications. Federated learning is a decentralized approach to training models across multiple local clients without the exchange of raw data from client devices to global servers. However, existing works focus on a static data environment and ignore continual learning from streaming data with incremental tasks. Federated Continual Learning (FCL) is an emerging paradigm to address model learning in both federated and continual learning environments. The key objective of FCL is to fuse heterogeneous knowledge from different clients and retain knowledge of previous tasks while learning on new ones. In this work, we delineate federated learning and continual learning first and then discuss their integration, i.e., FCL, and particular FCL via knowledge fusion. In summary, our motivations are four-fold: we (1) raise a fundamental problem called ''spatial-temporal catastrophic forgetting'' and evaluate its impact on the performance using a well-known method called federated averaging (FedAvg), (2) integrate most of the existing FCL methods into two generic frameworks, namely synchronous FCL and asynchronous FCL, (3) categorize a large number of methods according to the mechanism involved in knowledge fusion, and finally (4) showcase an outlook on the future work of FCL.
1 INTRODUCTION
The introduction frames FCL as a response to both streaming tasks and heterogeneous client data: learning new tasks can cause temporal forgetting, while aggregation can cause spatial forgetting. It presents FCL as knowledge fusion and previews experiments, two frameworks, and a seven-category survey of methods.
1 INTRODUCTION
- Privacy, data silos, and centralized-training constraints motivate collaborative learning that keeps private data local while clients upload models to a server.The introduction also notes that large-scale data transfer can be impractical, including for bandwidth-limited mobile devices.Source: §1 · p. 1
- Learning new tasks from local streams can cause TemporalCF, while differences among client data can cause SpatialCF when aggregation overwrites task-relevant parameters.SpatialCF is introduced for cases where the aggregated global model may perform worse than local models on their local test sets.Source: §1 · pp. 1–2
- FCL must address forgetting on both sides: clients retain earlier-task knowledge, while the server fuses heterogeneous local models, and SpatialCF may exacerbate TemporalCF.After a task, the server aggregates local models; clients then use the global model as the foundation for learning the next task.Source: §1 · p. 2
- The survey treats FCL training as fusion of previous- and current-task knowledge and heterogeneous client knowledge, using extracted knowledge to reduce privacy leakage and communication costs.The passage notes that uploading entire parameters or gradients may leak privacy and slow aggregation because of communication cost.Source: §1 · p. 2
- The paper defines two evaluation metrics and reports extensive experiments on the effects of spatial-temporal catastrophic forgetting in FCL.It also proposes the spatial-temporal forgetting problem as a fundamental FCL problem.Source: §1 · p. 2
- The proposed synchronous and asynchronous FCL frameworks are intended to integrate most existing methods and provide pipelines for future research.The paper also previews an outlook on future FCL research.Source: §1 · p. 2
- The survey categorizes FCL methods by seven knowledge-fusion forms: rehearsal, clustering, regularization, parameter isolation, dynamic architecture, prototype, and knowledge distillation.The introduction positions knowledge fusion as the organizing basis for its survey of existing methods.Source: §1 · p. 2
2 FEDERATED AND CONTINUAL LEARNING: AN OVERVIEW
The overview presents federated learning as collaborative training across clients without sharing raw data, while noting privacy risks in shared gradients and heterogeneity in client data. It connects continual learning’s task-sequence problem to federated learning’s client-level fusion problem as motivation for FCL.
2.1 Federated Learning
- Federated learning lets clients collaboratively train a shared model without sharing raw data, but shared gradients can still expose training data.The passage describes attacks that reconstruct original data or labels from gradients and uploaded knowledge.Source: §2.1 · p. 2
- The overview distinguishes HFL, VFL, and FTL by whether clients overlap in feature space, sample space, or both.HFL shares similar feature spaces, VFL similar sample spaces, and FTL has little overlap in either.Source: §2.1 · pp. 2–3
- Client data heterogeneity can make local models heterogeneous and the aggregated global model perform worse than some local models, a phenomenon the paper calls spatialCF.The text contrasts spatial unavailability of data across clients with temporal unavailability in continual learning.Source: §2.1 · p. 3
2.2 Continual Learning
- Continual learning aims to integrate knowledge from earlier and current tasks because sequential training can overwrite parameters and impair performance on previous tasks.The overview names regularization, parameter separation, dynamic architecture, replay, and distillation as approaches to mitigate forgetting.Source: §2.2 · p. 3
- The paper links continual learning’s sequential fusion across time with federated learning’s parallel fusion across clients, motivating FCL for heterogeneous task sequences and upload times.It also notes that combining CL with FL can reduce data storage and retraining costs when adding classes and reduce gradient-update leakage risk.Source: §2.2 · p. 3
- Prior adaptations include FedCurv’s shared-optimum penalty and FedCL’s server-estimated parameter importance weights, which incur at least twice the extra communication costs described for FedCurv.The overview also cites work adapting concept-drift methods to FL and examining the connection between concept drift and Non-IID data.Source: §2.2 · p. 3
3 SPATIAL-TEMPORAL CATASTROPHIC FORGET-
FedAvg experiments examine spatial and temporal forgetting in federated continual learning, where local models learn task sequences and a global model aggregates their updates. The results show that task-distribution differences are linked to temporal forgetting, while client-distribution differences are linked to spatial forgetting.
3.1 Problem Definition
- FCL faces temporal forgetting in local models as they learn new tasks and spatial forgetting when aggregation makes the global model perform worse on local data.The section defines spatial-temporal forgetting as average loss over all tasks seen by a communication round.Source: §3.1 · pp. 3–4
3.2.1 Settings
- The experiments vary client data across four scenarios, from shared classes and consistent distributions to clients with non-overlapping class sets.They also divide each client’s task sequence into three settings.Source: §3.2.1 · p. 4
3.2.2 Datasets
- Twelve CIFAR-100 experiments use three clients, one server, and three sequential tasks per client, with IID, Non-IID, or class-incremental task distributions.Each task’s data is split 7:3 into training and test sets, and evaluation uses test data from all tasks seen so far.Source: §3.2.2 · pp. 4–5
3.2.4 Experimental Results and Analysis
- Class-incremental tasks reduce temporal knowledge retention to zero, while identical client and task distributions avoid spatial-temporal forgetting and can improve performance.Spatial retention is highest in Scenario A, especially for class-incremental tasks; Scenario D has the lowest spatial retention, particularly with IID tasks.Source: Figure 1 · p. 5, §3.2.4 · p. 6
3.2.3 Metrics
- The section introduces temporal and spatial knowledge-retention metrics to assess forgetting, including retention of old-task knowledge locally and local-specific knowledge after aggregation.The temporal metric compares accuracy on the first task before and after subsequent task learning; the spatial metric measures local-specific knowledge retained by the aggregated global model.Source: §3.2.3 · p. 5
4 FEDERATED CONTINUAL LEARNING
FCL extends federated learning to task sequences, aiming to preserve prior-task knowledge locally while fusing heterogeneous knowledge globally. The section distinguishes shared-order and distinct-order settings, including federated class-incremental learning.
4 FEDERATED CONTINUAL LEARNING
- FCL must address local temporal forgetting while preventing server-side spatial forgetting when heterogeneous client models are fused.Its objective is to retain previous-task knowledge locally and fuse client knowledge without degrading local test performance.Source: §4 · p. 6
4.1 Synchronous FCL
- Synchronous FCL gives all clients the same task sequence; CFeD uses surrogate-data distillation and splits clients between learning new tasks and reviewing old ones.The setting is restrictive because the clients’ task sequences must align closely.Source: §4.1 · p. 6
4.2.1 Federated Task-Incremental Learning
- Asynchronous FCL allows distinct client task orders; ideal fusion incorporates each completed task without waiting, although most existing methods still aggregate local models.Different task sequences make this setting more practical but complicate spatial forgetting, requiring knowledge to be layered and fused separately.Source: §4.2.1 · p. 7, Figure 3 · p. 8
4.2.2 Federated Class-Incremental Learning.
- Federated class-incremental learning is a challenging asynchronous setting in which clients learn growing class sets and the global model should distinguish their union.The survey describes it as a common real-world problem for incrementally learning object classes, with only a few papers then available.Source: §4.2.2 · p. 8
5 KNOWLEDGE FUSION IN FCL
The section frames FCL as knowledge fusion on both the server and client sides, then organizes fusion methods by the forms of knowledge they use. It reviews rehearsal approaches that preserve or use past-task information in different ways.
5 KNOWLEDGE FUSION IN FCL
- The survey groups FCL knowledge-fusion methods into seven forms: rehearsal, clustering, all parameters, parameter or layer isolation, dynamic architecture, prototypes, and knowledge distillation.The review notes that some included techniques originate in traditional federated learning but can extend to FCL.Source: §5 · p. 9
5.1 Rehearsal
- Rehearsal methods retain exemplars from previous tasks and replay them during new-task training; FedPMR also stores past probability outputs for alignment,,.FedPMR’s Probability Distribution Alignment module maintains a small replay buffer of exemplars and their original probability outputs.Source: §5.1 · p. 9
- Shared replay buffers can address cross-client data heterogeneity, but they raise privacy concerns; pseudo-rehearsal generates data to simulate earlier task distributions.The survey reports Laplace noise for shared exemplars and identifies reduced memory needs and stronger privacy protection as advantages of pseudo-rehearsal.Source: §5.1 · p. 9
- FedKNOW integrates compact model-weight knowledge from signature tasks with the updated global model to limit forgetting and negative transfer [67].Signature tasks are the new task’s most dissimilar tasks among the client’s past tasks.Source: §5.1 · p. 10
- GradMA treats exemplars as inequality constraints and uses quadratic programming to correct local and global update directions iteratively.Clients use prior-round local and global gradients plus current parameter differences; the server uses accumulated participant gradients as constraints.Source: §5.1 · p. 10
5.2 Clustering
Clustering in FCL groups clients or tasks by similarity, chiefly to personalize models for heterogeneous client needs. The section contrasts client-level adaptation with group-level training and discusses its trade-offs.
5.2 Clustering
- FCL clustering can personalize at the client level by adapting a shared global model locally, or at the group level by training multiple models for clients with similar task distributions.A single global model may not meet every client’s specific requirements.Source: §5.2 · p. 10
- Different or even opposite outputs for the same input motivate multiple global models, and the survey says group-level personalization may have further potential.Source: §5.2 · p. 10
- Clustering clients by task similarity can exclude low-similarity uploads from a global model; GradMFL instead clusters Non-IID data into IID groups and uses hierarchical gradient memory [71],.The cited clustering approach reports a more effective resulting global model after excluding low-similarity client knowledge.Source: §5.2 · p. 10
- Separating local from global knowledge can reduce communication costs and preserve privacy, while multiple group models increase computation [20],.The survey says grouping can better aggregate similar knowledge and protect against a series of attacks.Source: §5.2 · p. 10
5.3 All Gradients/Parameters
This section contrasts parameter aggregation with regularization approaches to knowledge fusion. Regularization can help preserve past knowledge, but its effectiveness is limited when tasks differ substantially.
5.3 All Gradients/Parameters
- FedAvg averages local updates by data amount, but parameter averaging does not alleviate forgetting, integrate local and global knowledge, or prevent potential privacy leakage.Under class-incremental challenges, performance would significantly decrease.Source: §5.3 · p. 10
5.3.1 Regularization
- Regularization constrains parameter-update directions to make past knowledge less likely to be forgotten, and FCL can apply terms during global aggregation to preserve client knowledge.The section links spatial and temporal forgetting as targets for regularization-based approaches.Source: §5.3.1 · p. 10
- FedCurv [48] uses diagonal Fisher information to weight a client-side penalty toward a shared optimum, addressing statistical heterogeneity but leaving temporal catastrophic forgetting unsolved.It follows EWC.Source: §5.3.1 · pp. 10–11
- FedSI [81] combines local cross-entropy with a continual-learning loss that controls divergence between local models, guiding the combined global model toward a global optimum.The method integrates synaptic intelligence with federated learning.Source: §5.3.1 · p. 11
- FedCL estimates importance weights on a server using a proxy dataset and distributes them to clients, avoiding FedCurv’s client-side exchanges that incur at least twice the extra communication cost.The weights restrict subsequent local training.Source: §5.3.1 · p. 11
- Regularization works more efficiently on similar tasks and cannot discriminate classes across tasks in class-incremental settings, motivating more complex strategies for spatial-temporal fusion.Source: §5.3.1 · p. 11
5.4 Parameter Isolation
Parameter isolation limits what clients share and separates local isolation from global knowledge fusion. The section surveys attention-based aggregation and parameter decomposition as mechanisms for that fusion.
5.4 Parameter Isolation
- FCL parameter isolation partitions parameters to limit exposure from uploading all gradients or parameters, then combines local isolation with global fusion of selected information.Local isolation can use attention or parameter decomposition.Source: §5.4 · p. 11
5.4.1 Attention
- Layer-wise soft attention can select client-model weights for aggregation; FedAtt [88] treats the server as query and clients as keys to balance local-model contributions.Attentive aggregation generally operates on model weights rather than input-data flow.Source: §5.4.1 · p. 11
- A class-incremental approach combines dual feature attention with federated aggregation weights, while Self-Attention and Total Attention address current-task and all-observed-class forgetting, respectively.The dual-attention model identifies feature significance and reduces noise interference.Source: §5.4.1 · pp. 11–12
- FedMask shares only locally updated masks, which the server aggregates and applies to global-model initialization to fuse global knowledge while retaining local knowledge.The approach uses partial initialization to preserve local parameters.Source: §5.4.1 · p. 12
5.4.2 Parameter Decomposition
- FedPER [95] keeps client-specific final layers while sharing earlier layers, whereas VIRTUAL [96] combines private task-specific parameters with shared parameters and a server posterior.Source: §5.4.2 · p. 12
- FedWeIT decomposes weights into global and sparse task-specific parts, combining selected parameters across clients to mitigate incompatible-task interference and support positive knowledge fusion.Sparse selection also reduces interference between clients and facilitates efficient communication.Source: §5.4.2 · p. 12
- FedSeIT [100] selects relevant historical task-adaptive parameters using encoded representations to assess domain overlap, while separating dense local-base, sparse task-specific, and sparse-mask parameters.The survey identifies it as the first application of FCL to NLP.Source: §5.4.2 · p. 12
5.5 Dynamic Architecture
Dynamic architecture methods change network connections rather than continually adding parameters, while related designs connect shared and client-specific features in distinct ways.
5.5 Dynamic Architecture
- NetTailor-based FCL [101] uses a common backbone with task-specific blocks between layers, allowing links to bypass selected backbone layers.Source: §5.5 · p. 12
- CHFL separates common and unique features into two columns: common features train jointly, while unique features train locally through lateral connections without disrupting federated training.Source: §5.5 · pp. 12–13
- In FCL, dynamic architecture changes network connections rather than continually increasing parameter counts, unlike traditional continual-learning dynamic networks that add parameters for new tasks.Source: §5.5 · p. 13
5.6 Prototype
Prototype-based FCL exchanges compact class representations between clients and a server. The section covers methods for mitigating local and global forgetting, as well as aligning incremental tasks and heterogeneous client models.
5.6 Prototype
- Prototype-based methods [105]– exchange compact class vectors through the server, which merges and redistributes prototypes for knowledge transfer with low communication and memory costs.The surveyed works focus on class-incremental FCL.Source: §5.6 · p. 13
- GLFC combines class-aware gradient compensation for local class-imbalance forgetting with semantic-relation distillation and proxy-server selection of an old global model.Perturbed prototype samples from new classes support the proxy server’s model evaluation and selection.Source: §5.6 · p. 13
- Li et al. identify task overlap and heterogeneous client classifiers as issues in GLFC, proposing task-alignment tables and pre- or post-alignment of output layers.Full-covered, semi-covered, and not-covered scenarios distinguish client-side incremental tasks.Source: §5.6 · p. 13
- LGA [106] addresses differing old-class forgetting speeds with category-balanced gradient compensation and semantic distillation, while prototype augmentation helps select an old global model.Source: §5.6 · p. 13
- Federated reconnaissance [105] uses prototypes to communicate knowledge of incoming and previously observed classes, without relying on gradients for knowledge transfer.Clients share new-class knowledge with a central server, which merges and redistributes it.Source: §5.6 · pp. 13–14
5.7 Knowledge Distillation
The survey presents knowledge distillation as transferring a trained model’s knowledge to a student and describes several FCL implementations. These include using client and server teachers, distilling global knowledge for locally unavailable classes, and using unlabeled public data.
5.7 Knowledge Distillation
- Knowledge distillation transfers knowledge from a trained teacher model to a student, and is used in FCL methods.The teacher guides the student toward the same input-to-output mapping.Source: §5.7 · p. 14
- A class-semantic relation distillation loss maintains consistent inter-class relations across tasks [107].Source: §5.7 · p. 14
- In class-incremental FCL, a client’s past model teaches its current model, while the global model transfers knowledge from other clients.The past model supports proficiency on previously learned tasks; the global model is described as enhancing general features and mitigating overfitting on new tasks.Source: §5.7 · p. 14
- FedKL combines label-based classification supervision with global-model distillation for classes unavailable locally.The classification term uses softmax-cross-entropy; the knowledge-maintaining term uses logistic regression loss.Source: §5.7 · p. 14
- FCCL [113] uses distillation with unlabeled public data and self-supervised learning to provide inter- and intra-domain information without leaking privacy.The passage also describes its heterogeneous models as attaining generalizable representations with efficient communication.Source: §5.7 · p. 14
5.8 Summary
The survey compares knowledge-fusion methods by how knowledge is represented and transferred. It highlights trade-offs: methods that select or group relevant knowledge can avoid fusing irrelevant information, while rehearsal and prototype approaches face privacy and sample-space constraints.
5.8 Summary
- Regularization constrains parameter updates toward a common optimum, but large or opposing task differences can worsen performance.Source: §5.8 · p. 14
- Parameter isolation fuses relevant knowledge to improve global-model generalization, while clustering groups similar clients into multiple global models to avoid irrelevant knowledge.Source: §5.8 · p. 14
- Dynamic architectures can accommodate more knowledge but demand more careful design; distillation transfers knowledge through model inputs and outputs and may fail with unrelated teachers.Source: §5.8 · p. 14
- Rehearsal replays past data, which can create privacy risks, while its benefit for another client’s training is unclear.Source: §5.8 · p. 14
- Prototype methods compress data into sample-space prototypes, so applying them in FCL requires a consistent sample space.Source: §5.8 · p. 14
- Table 4 summarizes the knowledge-fusion methods used in the papers discussed.Source: §5.8 · p. 14, Table 4 · p. 15
6 FUTURE WORK AND DISCUSSION
The discussion outlines FCL frameworks, knowledge-fusion categories, and research directions spanning efficiency, trustworthy collaboration, and applications. It also reports experiments using FedAvg to examine spatial-temporal catastrophic forgetting and notes that FCL remains an emerging area.
FCL problem
- The survey defines spatial-temporal catastrophic forgetting as a fundamental FCL problem and reports experiments using FedAvg to examine its impact on global-model performance.Source: §6 · p. 15
Frameworks and knowledge fusion
- The survey summarizes synchronous FCL, focused on local-model aggregation, and asynchronous FCL, which shifts toward knowledge fusion, then classifies fusion methods by knowledge form.Source: §6 · p. 15
Trustworthy FCL
- Trustworthy FCL must balance privacy and utility alongside efficiency, robustness, fairness, incentives, and interpretability; the survey proposes Pareto optimization as a research direction.Source: §6 · p. 15
Delayed Decision-Making
- No study has yet incorporated three-way decision-making into FCL, despite its use in continual learning to delay decisions on uncertain samples.Source: §6 · p. 15
Rational Collaboration
- Rational collaboration is more complex in FCL because old and new tasks are heterogeneous, and can affect model performance.Source: §6 · pp. 15–16
Convergence Time
- FCL research should address convergence time, which can be prolonged by federated communication and rehearsal-based methods, rather than focusing only on accuracy.Source: §6 · p. 16
Recommendation System
- Federated continual recommendation could address changing user preferences and item catalogs with up-to-date recommendations.Source: §6 · p. 16
Large Language Model
- LLMs are being explored for federated pretraining and continual learning, but research combining FCL with LLMs remains limited [138].Source: §6 · p. 16