TARGET: Federated Class-Continual Learning via Exemplar-Free Distillation

1. Introduction

• FCCL addresses dynamically added classes in federated learning, where transferring a model to new tasks can degrade performance on previous classes through catastrophic forgetting.Source: §1 · p. 1
• Existing FCCL methods may require surrogate datasets or historical client data, which can be difficult to obtain or retain in privacy-sensitive settings.The introduction asks how to mitigate forgetting without storing clients’ private data or using additional datasets.Source: §1 · pp. 1–2
• TARGET transfers old-task knowledge from the previous global model and uses generated data to simulate each client’s global data distribution.The method is presented as mitigating forgetting without compromising client data privacy.Source: §1 · p. 2
• 36.31% accuracy on five-task CIFAR-100, about 6% higher than the best baseline, for TARGET.Source: §1 · p. 2

2.1. Federated Continual Learning

• Federated continual learning methods address client-wise forgetting, inter-client interference, and communication efficiency through parameter decomposition, drift adaptation, distillation, gradient compensation, or selective transfer.Examples include FedWeIT [58], CDA-FedAvg [14], CFeD, GLFC [12], and FedSeIT [5].Source: §2.1 · p. 2

2.2. Continual Learning

• Continual-learning approaches include regularization [23] [61] [27], replay [35, 45, 49, 46, 44, 8], architecture changes [60] [55], and data-free methods [57] [50].The cited data-free methods are framed as useful when storing old-task data is infeasible or undesirable, including privacy-sensitive applications.Source: §2.2 · pp. 2–3

3.1. Problem Definition

• FCCL learns nonoverlapping new-class tasks over time; clients cannot share raw data and can access only current-task data during its training period.The setting assumes heterogeneous clients with skewed label distributions.Source: §3.1 · p. 3
• Because clients cannot access old-task data, the objective pairs current-class classification with limiting changes to previously learned classes.The regularization term encourages the new model parameters to remain close to the previous model parameters.Source: §3.1 · p. 3

3.2. Heterogeneous Data Exacerbates Forgetting

• Greater non-IIDness exacerbates forgetting: accuracy declines even under IID partitions, and performance is worse as client data become more imbalanced.The CIFAR-100 experiment partitions 100 classes into five tasks across five clients; lower Dirichlet values indicate greater imbalance.Source: Figure 1 · p. 3, §3.2 · p. 4

3.3. Alleviating Forgetting via Global Information

• Exemplars improve performance on current and previous tasks, with global exemplars achieving higher accuracy than local exemplars, but selecting them raises privacy concerns.Global exemplars aggregate and redistribute client data, whereas local exemplars retain prior local data.Source: Figure 2 · p. 4, §3.3 · pp. 4–5
• FedAvg reaches 0% accuracy on Tasks 1, 2, and 3, while FedLwF improves accuracy on Tasks 1–4 and final accuracy under the reported partitions.The comparison evaluates previous-task accuracy after training on all five tasks.Source: §3.3 · p. 4, Table 1 · p. 4
• TARGET inverts the global model trained on task k−1 to synthesize globally distributed data for training task k.The pipeline combines synthesized data with real current-task data for local model updates.Source: Figure 3 · p. 5

4.1. Overview

• TARGET inverts the previous global model to synthesize old-task data, then combines it with current-task real data for local updates.The method uses global information without storing real client data.Source: §4.1 · p. 5

4.2. Server Side: Synthesizing Data for Old Tasks

• The server trains a generator to model the global teacher’s distribution and a student to distill the teacher’s knowledge using synthetic data.The student’s performance is used to improve the generator; after training, only the synthetic dataset is retained and sent to clients.Source: §4.2 · pp. 5–6
• Boundary support loss steers generated samples toward teacher–student decision disagreements, while batch-normalization loss aligns synthetic data with batch statistics.The authors state that the boundary-support loss is intended to improve transferability and that batch-normalization loss improves conformity to batch-normalization statistics.Source: §4.2 · p. 6
• The generator objective combines classification, diversity, and batch-normalization losses, with λ1 and λ2 weighting the latter two terms.Source: §4.2 · p. 6

4.3. Client Side: Update with Global Information

• Clients train on current-task data with cross-entropy and use KL distillation on synthetic data to transfer previous-task knowledge from the global teacher.The hyperparameter α controls the strength of regularization for previous tasks.Source: §4.3 · p. 6

5.1. Experimental Settings

• Experiments use CIFAR-100 and Tiny-ImageNet, divide classes equally into 5 or 10 tasks by default, and use ResNet18 as the classification backbone.Comparisons include sequential fine-tuning, FedWeIT, FedEWC, FedLwF, and FediCaRL.Source: §5.1 · pp. 6–7

5.2. Experiments on CIFAR-100

• 36.31% average accuracy in 5-task IID CIFAR-100 is about 6% above FedLwF; TARGET also outperforms competing baselines across incremental tasks.The experiments report that TARGET maintains the highest average accuracy and lowest forgetting as the task count rises to 10.Source: Table 2 · p. 7, §5.2 · p. 7, Figure 4 · p. 7

5.3. Experiments on Tiny-ImageNet

• On 5-task Tiny-ImageNet, TARGET exceeds FedLwF in average accuracy across all tested partitions and has lower forgetting in IID and non-IID settings.In the most challenging NIID(0.05) setting, its average accuracy is approximately 3% higher.Source: Figure 5 · p. 8, §5.3 · p. 8

5.4. Comparison with Exemplar-based Method

• With 2k synthetic examples, TARGET matches iCaRL with 1k real examples but does not surpass iCaRL with 2k real examples.TARGET does not store real training data, and increasing its synthetic memory from 2k to 3k improves accuracy.Source: Figure 6 · p. 8, §5.4 · p. 8

5.5. Analysis of Our Method

• On five-task CIFAR-100, 8k synthetic examples alleviate forgetting, while increasing the volume to 12k or 16k yields no significant improvement.The authors caution that data volume alone does not ensure effectiveness; synthetic-data quality and relevance also matter.Source: Figure 7 · p. 9, §5.5 · p. 9
• Visualized synthetic images from Tiny-ImageNet differ visually from real training data, though the authors say they may help the student approach teacher performance rapidly.The visualization follows learning of the penultimate task in a five-task setup.Source: Figure 8 · p. 9, §5.5 · p. 9
• For two-task CIFAR-100, α values between 10 and 15 provide a good balance between accuracy on new and old tasks.Larger α favors backward transfer, while smaller α favors forward transfer.Source: §5.5 · pp. 8–9
• Using synthetic data, server-side distillation quickly brings the student model close to the global model without accessing clients’ private training data.Source: Figure 9 · p. 9, §5.5 · p. 9

• TARGET uses global knowledge to alleviate catastrophic forgetting in FCCL without requiring additional datasets or data from previous tasks.Source: §6 · p. 9