PaperlayerView exact revision on arXiv

Source-linked AI summary

ReSCENE: Server-Side Replay for Structural Mitigation of Catastrophic Forgetting in Federated Continual Learning

Sungmin Kang, Zhengzhong Tu, Sunwoo Lee

arXiv:2609.38833v1cs.LG

TL;DR

Federated continual learning must learn arriving tasks while retaining earlier-task knowledge, but existing approaches can constrain new learning or burden clients. ReSCENE replays condensed surrogates from earlier tasks on the server alongside current-task surrogates. Across three benchmarks, it reports the strongest accuracy among seven baselines, with lower client computation and upload.

  • Problem

    Federated continual learning must integrate new tasks while retaining earlier-task knowledge, but existing anti-forgetting methods can hold back new learning or burden resource-constrained clients.

    Source: §1 · p. 1

  • Method

    ReSCENE has clients upload condensed surrogates and replays earlier-task surrogates on the server alongside those of the current task.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    ReSCENE achieves the strongest accuracy among seven FCL baselines across the reported benchmarks; on CIFAR-10 at β=0.1 it leads Re-Fed by 31.1 AA points.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    ReSCENE maintains its reported advantage with larger client populations and larger models while requiring substantially lower client computation and upload.

    Source: §1 · p. 2, §6 · p. 9

    Surrogate quality depends on the dataset condensation method.

    Source: §6 · p. 9

Abstract

from arXiv · show

Federated continual learning must integrate new tasks over time without losing earlier-task knowledge. Most existing methods attach an anti-forgetting mechanism to the client-trained, server-aggregated loop of federated learning, which holds back new learning to preserve earlier knowledge and burdens resource-constrained clients. We propose ReSCENE, which structurally mitigates catastrophic forgetting by having each client upload a small condensed surrogate of its local data while the server keeps the surrogates of past tasks and trains the global model on them together with the current task surrogates. For efficient server memory, we introduce temporal herding, which selects the more recent surrogates from the pool accumulated over a task into a compressed buffer. Our study provides a theoretical analysis showing that this buffer can represent the original task data more closely than full accumulation of all surrogates. Across CIFAR-10, CIFAR-100, and TinyImageNet, ReSCENE achieves the strongest accuracy over seven baselines, by up to $31.1$ points of average accuracy, while requiring as little as $0.11\times$ of the client computation and up to $179\times$ less upload than the model-update baselines. ReSCENE further demonstrates its effectiveness when scaled to larger client populations and larger models while remaining efficient, which makes it a practical method for federated continual learning.

1 Introduction

ReSCENE moves forgetting mitigation to the server, which replays earlier-task surrogates alongside current-task surrogates. It introduces temporal herding to bound server memory and reports strong benchmark accuracy with lower client costs.

1 Introduction

  • Existing FCL methods preserve earlier-task knowledge but may constrain new learning or add memory and computation burdens to resource-constrained clients.As task sequences grow, earlier-task performance can continue to degrade.Source: §1 · p. 1
  • The server retains earlier-task surrogates and trains on them with current-task surrogates, structurally mitigating forgetting by optimizing old and new tasks together.Unlike prior surrogate-aggregation FL, ReSCENE reuses accumulated surrogates as a replay source.Source: §1 · pp. 1–2
  • Temporal herding compresses each task’s surrogate pool into a bounded buffer centered on recent rounds, which can represent original task data more closely than the full pool.Later-round surrogates represent the task more faithfully as the global model and surrogates co-evolve.Source: §1 · p. 2, Figure 1 · p. 2
  • ReSCENE achieves the strongest performance among seven FCL baselines, using as little as 0.11× FedAvg client computation and up to 179× less upload than a model update.It reaches 67.96% AA / 73.53% AIA on CIFAR-10 and 49.80% / 57.71% on CIFAR-100.Source: §1 · p. 2

2 Related Work

Prior FCL approaches add regularization, distillation, or replay to federated learning, each with trade-offs. ReSCENE differs from static surrogate-aggregation FL by retaining and replaying surrogates across tasks.

Anti-forgetting in FCL

  • Regularization limits updates important to earlier tasks, while distillation matches a previous-task teacher; both approaches can hinder learning new knowledge.Regularization leaves little room for new knowledge as tasks continue, and a stale teacher pulls the model toward earlier states.Source: §2 · p. 2
  • Exemplar replay preserves earlier knowledge by training on stored raw data, but adds computation and consumes scarce client memory.Generative replay avoids storing raw data, but generated samples can drift from the real distribution in heterogeneous federated settings.Source: §2 · pp. 2–3

Surrogate-aggregation FL

  • Static surrogate-aggregation methods train on each uploaded surrogate once and discard it, whereas ReSCENE retains surrogates for replay as later tasks arrive.FedDM and FedAF study static training with fixed local data.Source: §2 · p. 3

3 Method

ReSCENE trains the global model with server-side replay of condensed client surrogates, compresses completed tasks into buffers, and corrects class imbalance during server training. Its theoretical analysis gives a condition under which a herded buffer represents a class mean more closely than full accumulation.

3 Method

  • ReSCENE has clients upload condensed local-data surrogates and trains the global model on current-task surrogates together with earlier-task buffers.At each task’s end, temporal herding compresses its pool into a bounded buffer.Source: §3 · p. 3

3.1 Problem Setup and Notation

  • The setup models N clients and T tasks of R rounds, with each task introducing a disjoint set of new classes; the server tracks task pools and replay buffers.The global model and its feature extractor are denoted fθ and ϕθ.Source: §3.1 · p. 3

3.2 Surrogate Aggregation for Federated Continual Learning

  • Clients condense local data into small per-class synthetic sets, and the server jointly trains on surrogates from seen tasks to learn boundaries across their classes.Task-split optimization does not train cross-task boundaries, whereas ReSCENE’s pooled training covers every seen class in one objective.Source: §3.2 · pp. 3–4

3.3 Temporal Herding

  • Temporal herding selects a bounded task buffer from the full pool to approximate the target formed by recent-round surrogates, which are more faithful as model and surrogates co-evolve.The target uses the recent ⌈λR⌉ rounds; selected candidates are chosen to bring their mean close to that target.Source: §3.3 · p. 4

3.4 Resolving Class Imbalance in Server-Side Training

  • Because current-task surrogates accumulate while earlier classes are capped at B, the server pool skews toward new classes; ReSCENE uses sampling and logit adjustment to address the imbalance.The sampling exponent α controls class sampling, and logit adjustment uses the surrogate-pool prior.Source: §3.4 · pp. 4–5

3.5 Theoretical Analysis of Temporal Herding

  • Under its stated condition, temporal herding’s buffer represents the real class mean more closely than full accumulation, despite retaining only B surrogates per class.The condition requires removed stale bias to exceed deviation and approximation error; the analysis assumes directional coverage and coherent stale bias.Source: §3.5 · p. 5

4 Experiments

Across the FCIL benchmarks, ReSCENE leads the baselines on most accuracy measures and best retains earlier-task accuracy, balancing performance on old and recent tasks.

4.1 Experimental Setup

  • The FCIL evaluation uses CIFAR-10, CIFAR-100, and TinyImageNet, with 20 clients, 10 participants per round, 20 rounds per task, and three-seed AA/AIA reporting.All methods use ResNet-18; AA measures accuracy after the final task, while AIA averages accuracy across tasks.Source: §4.1 · pp. 5–6

4.2 Main Results

  • ReSCENE attains the highest AA in all nine settings and highest AIA in seven; at β=0.1, its AA lead over Re-Fed is 31.1 points on CIFAR-10.On CIFAR-100, it leads FedCBDR by 8 to 11 AA points at every β; Table 1 reports AA and AIA across the benchmarks.Source: Table 1 · p. 6, §4.2 · p. 7
  • Both SA-FL baselines reach 0% final accuracy on every earlier task, while ReSCENE retains earlier-task performance by replaying stored surrogates.Their AA stays at 16% or below; Table 2 compares the SA-FL baselines and ReSCENE at β=0.1.Source: Table 2 · p. 7, §4.2 · p. 7

4.3 Forgetting Analysis

  • ReSCENE has the highest old-task accuracy on every benchmark and lies near the old-versus-recent accuracy diagonal, unlike baselines that generally fall below it.For CIFAR-10, the old group is T0–T1 and recent group T2–T4; the ten-task benchmarks split into five-task groups.Source: Figure 3 · p. 6, §4.3 · p. 7

5 Ablations and Analyses

ReSCENE retains strong accuracy as client populations and models grow, while temporal herding and reduced server training preserve performance with fewer resources.

5.1 Scalability to Larger Client Populations and Models

  • At N=100, ReSCENE reaches 72.13 AA, gaining 4.17 AA and 4.93 AIA points over N=20, with 0.25 MB uplink versus 44.73 MB.It uses 0.47× FedAvg’s computation; the table’s arrows show changes from N=20.Source: §5.1 · p. 7, Table 3 · p. 8
  • Across model sizes, ReSCENE leads the strongest baseline by 8.16, 18.81, and 8.79 AA points, with 1.23 MB uplink for every backbone.Its client computation is about 2.9% of FedCBDR’s and 18.8% of Re-Fed’s.Source: Table 3 · p. 8, §5.1 · p. 8

5.2 Herding-Buffer Ablation

  • Against full accumulation, ReSCENE trails by 1.23 AA points on CIFAR-10 but leads by 2.26 on CIFAR-100 and 6.07 on TinyImageNet.It holds 0.60 to 0.79× the surrogates and requires 0.69 to 0.78× the server computation.Source: §5.2 · p. 8, Table 4 · p. 9
  • With B=1000, temporal herding achieves the highest AA and AIA among five buffer strategies, improving on Latest-1K by 1.40 AA points.It selects from the whole surrogate pool toward a target defined by recent surrogates.Source: §5.2 · p. 8, Table 4 · p. 9

5.3 Server Training Budget Ablation

  • Reducing server epochs from 100 to 10 cuts server computation by 90%, while AA drops 8.33 points and AIA drops 3.32 points.At 10 epochs, ReSCENE still leads Re-Fed by 22.78 AA points.Source: §5.3 · p. 9

6 Conclusion

ReSCENE shifts forgetting mitigation to the server and uses temporal herding to compress earlier-task surrogates. Across three class-incremental benchmarks, it reports the strongest accuracy at lower client computation and uplink than the baselines, while retaining an advantage with more clients and larger models.

6 Conclusion

  • Across three class-incremental benchmarks, ReSCENE attains the strongest accuracy at a fraction of the baselines’ client computation and uplink, and keeps its advantage with more clients and larger models.The paper presents ReSCENE as a practical and efficient solution for federated continual learning.Source: §6 · p. 9
  • ReSCENE shifts forgetting mitigation to the server, retaining earlier-task surrogates and training on them alongside current-task surrogates.Temporal herding compresses each task’s surrogate pool into a bounded buffer centered on recent rounds.Source: §6 · p. 9
  • The quality of ReSCENE’s surrogates depends on the dataset condensation method, making better local-data representation an open direction.Source: §6 · p. 9

A Detailed Experimental Setup and Hyperparameters

The appendix specifies the benchmark settings and implementation choices, then reviews federated continual-learning approaches and their shared constraints. It covers regularization, distillation, architecture-based, and replay-based methods.

A Detailed Experimental Setup and Hyperparameters

  • CIFAR-10 uses five two-class tasks, CIFAR-100 ten ten-class tasks, and TinyImageNet ten 20-class tasks, with Dirichlet label heterogeneity at β ∈{0.1, 0.5, 1.0}.The common protocol uses 20 clients, 10 participants per round, 20 rounds per task, and ResNet-18; ReSCENE uses IPC 10 and a per-class buffer budget of 1000.Source: A Detailed Experimental Setup and… · p. 13

B.1 Anti-Forgetting in Federated Continual Learning

  • Regularization methods constrain updates using earlier-task importance, while distillation methods match a previous-task teacher; their stated drawbacks are accumulating constraints and a stale teacher.The distillation discussion also notes that class-incremental tasks and the teacher can have entirely different distributions.Source: §B.1 · p. 14
  • Architecture-based methods reserve task-specific parameters, but coordination across clients is difficult and accumulated parameters can shrink capacity for new tasks.Powder and PILoRA also require every client to hold a pre-trained backbone.Source: §B.1 · p. 14
  • Exemplar replay burdens clients with storage, selection, and local training, while generative replay can drift from earlier-class real-data distributions.The review distinguishes exemplar-based methods from generative methods that synthesize past-class data.Source: §B.1 · pp. 14–15

C Theoretical Analysis of Temporal Herding

The analysis compares temporal herding’s bounded buffer with full-pool alternatives by tracking surrogate features at the task-end model. Recent rounds define the target, while candidates remain eligible from the whole task pool.

C Theoretical Analysis of Temporal Herding

C.1 Setup

  • For each class, the setup indexes uploaded surrogates by communication round and defines a recent-round window using λ and R.The target window contains the recent ⌈λR⌉ rounds, with a fallback to the latest round if it is empty.Source: §C.1 · p. 15
  • Temporal herding forms its target from the recent candidates’ normalized task-end features, then greedily selects a buffer from the full candidate pool.Source: §C.1 · p. 15

C.2 Recent Surrogates Reduce Task-End Error

The theoretical analysis motivates recent-round targets, bounds greedy buffer approximation, and gives conditions under which temporal herding is closer to the task-end class mean than full-pool alternatives. The appendix also lists additional experiment topics.

C.2 Recent Surrogates Reduce Task-End Error

  • Recent-round surrogates reduce the staleness component of task-end error because their associated model states are closer to the task-end model.The bound assumes normalized feature means change smoothly along the within-task model trajectory.Source: §C.2 · p. 16

C.3 Temporal Herding Approximates the Target within a Bounded Error

  • Under directional coverage, greedy temporal herding keeps the finite buffer close to the recent target, with the approximation error shrinking as the buffer budget grows.Source: §C.3 · p. 17

C.4 When the Bounded Buffer Surpasses Full Accumulation

  • Under Assumptions 1 and 2, temporal herding’s buffer is strictly closer to the real class mean than full accumulation when the stated condition holds.The analysis says the condition compares removed stale bias against surrogate deviation and finite-buffer approximation error; with many rounds, the removed bias is expected to exceed those fixed terms.Source: §C.4 · p. 18

D Additional Experiments

  • Additional experiments report per-task accuracies, domain-incremental evaluation, replay-budget comparisons, temporal-herding hyperparameter sweeps, and server-objective ablations.Source: D Additional Experiments · p. 19

D.1 Detailed Per-Task Results

Per-task results show that ReSCENE retains earlier and recent tasks more evenly than the baselines, including under domain shift. It also outperforms replay baselines when their total replay-image budget is matched, while temporal herding balances accuracy and buffer size.

D.1 Detailed Per-Task Results

  • ReSCENE’s lowest per-task accuracy exceeds every baseline’s in all nine settings: 36.2–46.9% on CIFAR-10, 32.3–44.8% on CIFAR-100, and 19.9–21.6% on TinyImageNet.On CIFAR-10, its oldest-task accuracy is 75–78%, while no baseline exceeds 39%.Source: §D.1 · p. 19

D.2 Domain-Incremental Learning

  • ReSCENE attains the highest accuracy on each Office-31 domain—56.48% on Amazon, 65.66% on DSLR, and 76.58% on Webcam—and leads all methods with 66.24% AA and 56.10% AIA.The evaluation uses the Amazon, DSLR, and Webcam domains in sequence under one shared 31-class classifier.Source: §D.2 · p. 19, Table 6 · p. 19

D.3 Comparison under a Matched Replay Budget

  • With the same 10,000 replay images, ReSCENE exceeds each replay baseline by 17.2–48.3 AA points and 10.8–35.9 AIA points.Matching replay budgets does not close the gap; the comparison counts replay images held by the server and clients at the sequence’s end.Source: §D.3 · p. 20, Table 8 · p. 22

D.4 Additional Herding-Buffer Sweeps

  • λ=0.75 gives the highest AA and AIA among four tested temporal-target fractions; λ=0.25 reaches 66.92 AA and full-pool herding at λ=1 reaches 66.70 AA.A narrow recent window can be biased toward a few rounds, while the full pool includes stale early surrogates.Source: §D.4 · pp. 21–22
  • Increasing B from 500 to 1000 improves AA by 6.85 points; B=1000 stays within 1.5 AA points of larger budgets while holding 40% fewer surrogates than the whole pool.The bounded buffer is also more accurate than full accumulation on CIFAR-100 and TinyImageNet, consistent with Theorem 1.Source: §D.4 · p. 22

D.5 Ablation of Imbalance Resolution

On CIFAR-10, balanced sampling and logit adjustment each improve on natural training, and using both gives the strongest AA and AIA. ReSCENE’s design shifts most computation from clients to the server.

D.5 Ablation of Imbalance Resolution

  • Using both balanced sampling and logit adjustment gives the highest AA and AIA, 0.70 and 0.98 points above natural training, respectively.Each correction alone improves on natural training; sampling acts on batch composition, while logit adjustment corrects classifier bias.Source: §D.5 · p. 22, Table 10 · p. 22

E System Cost Analysis

  • ReSCENE’s clients condense surrogates while most computation takes place at the server, shifting the computational burden away from clients.The system-cost analysis therefore considers computation across both clients and server.Source: E System Cost Analysis · pp. 22–23

E.1 Client-Side Resource Accounting

ReSCENE reduces client-side computation, uplink, and persistent state relative to the weight-communicating methods, while shifting most computation to the server. Its surrogate uploads can be made differentially private, and server-side storage and replay require no further privacy budget.

E.1 Client-Side Resource Accounting

  • ReSCENE uses the least client computation and uplink on every benchmark and the smallest client state, tied with FedAvg.Client computation is 0.1× FedAvg on CIFAR-10 and 0.8× on CIFAR-100 and TinyImageNet; uplink is 0.25 MB on CIFAR-10 and 9.83 MB on TinyImageNet, versus 44.73–45.12 MB for other methods.Source: §E.1 · p. 23, Table 11 · p. 23

E.2 Total Computation across the Server and Clients

  • On CIFAR-10, ReSCENE spends 27.32 PFLOP total versus 52.64 PFLOP for TARGET, including 26.6 versus 38.2 PFLOP on the server and 0.72 versus 14.44 PFLOP on clients.Clients carry 2.6% of ReSCENE’s total computation; the server carries the remaining 97.4%.Source: §E.2 · p. 24, Table 12 · p. 24

F.1 Differential Privacy of the Surrogate Upload

  • With DP-SGD clipping each per-example surrogate-pixel gradient term and adding Gaussian noise, each round’s uploads, derived global model, and buffers are (ϵ, δ)-differentially private under Theorem 2’s condition.The proposition assumes surrogate initialization from random noise.Source: §F.1 · p. 25
  • Server-side storage, temporal-herding compression, and replay use only uploaded surrogates, so they raise no further privacy concern and the privacy budget is spent once, at upload time.The server does not access client data again for these operations.Source: §F.1 · p. 25

F.2 Empirical Robustness to Gradient Perturbation

Under the shared CIFAR-10 gradient-perturbation protocol, accuracy decreases as noise grows, with model-update baselines losing AA and AIA sharply. ReSCENE retains substantially more AA at the highest tested noise level.

F.2 Empirical Robustness to Gradient Perturbation

  • The comparison uses CIFAR-10 at β=0.5 with σ∈{0, 1, 3, 5} and C=5, and compares accuracy under the same perturbation rather than certifying a common (ϵ, δ).Model-update baselines use GroupNorm instead of BatchNorm at every σ, including σ=0.Source: §F.2 · pp. 25–26
  • At σ=5, FedAvg and FedCBDR drop to 0% on earlier tasks T0–T3, while Re-Fed retains only 8.10% on T0; all three keep about 50% only on the last task.FedAvg falls to 10% AA from σ=1 onward, and FedCBDR and Re-Fed reach a similar value from σ=3.Source: §F.2 · p. 26, Table 13 · p. 26
  • At σ=5, ReSCENE retains 55.13% AA, 81.2% of its 67.86% at σ=0, while accuracy remains spread over earlier tasks.The authors attribute this robustness to noise perturbing only uploaded surrogates, with no further noise added during server training.Source: §F.2 · p. 26
Loading 2609.38833v1…