PaperlayerView exact revision on arXiv

Source-linked AI summary

Task Relevance-Driven Prompt Expansion for Federated Continual Learning

Jiahua Dong, Wenqi Liang, Yang Cong, Yichen Li, Hongyuan Yu, Henghui Ding, Yulun Zhang, Salman Khan, Fahad Shahbaz Khan

arXiv:2610.34461v1cs.CV

TL;DR

Existing FCL methods assume the server knows task arrivals and class counts, and often overlook useful inter-task relationships. TRPE detects new tasks and estimates their class counts while using global task relevance to guide prompt use; it reports state-of-the-art performance across diverse class- and domain-incremental settings.

  • Problem

    Existing FCL methods assume prior server knowledge of new-task arrival times and category counts, while heterogeneous client distributions make that information unrealistic to obtain.

    Source: §1 · p. 2

  • Method

    TRPE detects new-task arrivals locally, estimates class counts with radius-adaptive DBSCAN, and uses global gradient-based task relevance to guide prompt expansion or reuse.

    Source: §1 · pp. 2–3

  • Results

    TRPE reports state-of-the-art performance across diverse class- and domain-incremental FCL settings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A soft orthogonality constraint on key and prompt gradients is used to balance learning new tasks and mitigating forgetting of old tasks.

    Source: §1 · p. 3

    The work focuses on class- and domain-incremental FCL, excluding task-incremental learning because task identity would severely compromise client privacy.

    Source: §3 · p. 3

Abstract

from arXiv · show

Federated continual learning (FCL) has attracted considerable interest by enabling local clients to continually acquire knowledge from new tasks. However, most FCL methods assume that the global server has prior knowledge of the arrival times and number of categories for new tasks, which is unrealistic in the real world. Furthermore, they neglect the importance of positive inter-task semantic relationships when learning new tasks and are significantly affected by irrelevant relationships from old tasks, thereby exacerbating catastrophic forgetting. To resolve the above challenges, we develop a novel Task Relevance-driven Prompt Expansion (TRPE) model. Specifically, to eliminate the reliance on prior knowledge of new tasks, we propose an out-of-distribution detector to identify the arrival of new tasks within local clients, along with a radius-adaptive DBSCAN to infer the quantity of categories in new tasks on the proxy server. Moreover, to explore positive inter-task relationships, we calculate the affinity between the original gradient for learning new task and its gradient orthogonal to the feature space of old tasks to evaluate semantic relevance between old and new tasks from a global perspective. Then, we use this global task relevance to determine the key-prompt pair for new tasks and identify highly related old tasks. After concatenating their prompts for training, we can leverage such positive inter-task relations to improve the continual learning of newly arriving tasks. To balance the learning of new tasks and the forgetting of old ones, a soft orthogonality constraint is applied to the local key and prompt gradients. Experiments spanning diverse datasets and backbones confirm the efficacy of our TRPE model.

1 INTRODUCTION

TRPE addresses unknown task arrivals and class counts in federated continual learning while using task relevance to exploit positive inter-task relationships and reduce forgetting. The paper reports state-of-the-art performance across diverse class- and domain-incremental settings.

1 INTRODUCTION

  • Existing FCL methods assume the server knows new-task arrival times and category counts, despite heterogeneous client distributions making this prior knowledge unrealistic.Source: §1 · p. 2
  • TRPE estimates global semantic relevance by comparing the new-task gradient with its component orthogonal to old-task feature spaces.Source: §1 · pp. 2–3
  • An out-of-distribution detector identifies new-task arrivals locally, while radius-adaptive DBSCAN estimates each new task's class count without prior knowledge.Source: §1 · p. 2
  • TRPE reports state-of-the-art performance across diverse class- and domain-incremental FCL settings.Source: §1 · p. 2
  • Global task relevance guides whether the server expands a key-prompt pair or reuses one from the most relevant previous task.Source: §1 · pp. 2–3
  • Prompts for new and highly related old tasks are concatenated for training, with soft orthogonality on key and prompt gradients balancing new learning and forgetting.Source: §1 · pp. 2–3

2 RELATED WORK

Related work spans federated learning, continual learning, and their combination in FCL. Existing FCL methods include rehearsal-based, rehearsal-free, and prompt-based approaches, but often neglect useful inter-task relationships.

2 RELATED WORK

  • Federated learning aggregates client updates while keeping data decentralized; McMahan et al. [38] constructed a global model by aggregating parameters learned across clients.Source: §2 · p. 3
  • Continual-learning methods use regularization, replay, architecture expansion, or representation learning, but some cannot transfer directly to FCL because they require privacy-sensitive task-arrival information.Source: §2 · p. 3
  • Some prompt-based FCL methods [3],, [37], help local clients collaboratively alleviate forgetting, but most existing approaches, [13],, [69] neglect intrinsic inter-task relationships that could reduce forgetting and improve learning on newly emerging tasks.Source: §2 · p. 3

3 PROBLEM DEFINITION

The problem is collaborative learning of consecutive tasks across distributed clients while preserving privacy and avoiding forgetting. The setting assumes unknown task arrivals and client data distributions, and focuses on class- and domain-incremental learning.

3 PROBLEM DEFINITION

  • FCL learns a sequence of tasks across distributed clients while preserving data privacy.Source: §3 · p. 3
  • The paper focuses on class- and domain-incremental settings because task identity in task-incremental learning would severely compromise client privacy.Source: §3 · p. 3
  • The server aggregates updated client models over communication rounds, and clients learn without storing exemplars from old tasks.Source: §3 · pp. 3–4
  • Task quantities, arrival times, and client data distributions are unknown in advance, and the global model is trained to learn tasks consecutively without forgetting.Source: §3 · p. 4

4 THE PROPOSED MODEL

TRPE addresses the lack of prior task information and harmful irrelevant inter-task relationships in federated continual learning by detecting new tasks and estimating their class counts, then using task relevance and stored prompts to guide learning under a soft orthogonality constraint.

4.1 Preliminary

  • In the preliminary prompt-based FCL setup, L2P [66] uses a key to retrieve its task’s prompt, and testing selects a prompt by matching the image features to stored keys.The paper uses L2P as an example of a setup that can also use other prompt technologies.Source: §4.1 · p. 4

4.2.1 How to Address Dependence on Prior Knowledge

  • TRPE identifies new-task arrivals locally with an out-of-distribution detector and estimates new-task class counts at a proxy server with radius-adaptive DBSCAN.The method is designed to avoid requiring the server to know task arrival times or category counts in advance.Source: §4.2.1 · pp. 4–5
  • The proxy server adapts DBSCAN’s radius to prototype distances, reducing the impact of noisy prototypes and outliers from heterogeneous client data when estimating category counts.The method sets the minimum cluster size to 3 to mitigate class imbalance among prototypes.Source: §4.2.1 · p. 5

4.2.2 How to Explore Positive Inter-Task Relationships

  • The proxy server measures global task relevance from prompt gradients and uses it to guide whether to expand or reuse a key-prompt pair and which old prompts clients concatenate for training.This is intended to exploit positive semantic relationships while ignoring irrelevant ones.Source: §4.2.2 · p. 6
  • For each new task, the proxy server constructs an expanded feature space by adding orthogonal bases derived from global class prototypes after projecting out the previous task’s feature space.It obtains the new bases through a low-rank approximation of the projected prototype representation matrix.Source: §4.2.2 · p. 6
  • TRPE scores task relevance using affinity between a global prompt gradient and its projection orthogonal to an old-task feature space, with an elastic threshold to account for data heterogeneity.The threshold compares the gradient with its projection onto the new task’s orthogonal feature space.Source: §4.2.2 · pp. 6–7
  • The global server selects prompts from stored tasks for concatenation with the new task’s prompt, while freezing updates to the selected prompts during local training.The combined prompt replaces the original prompt for learning the new task.Source: §4.2.2 · p. 7

4.3 Optimization

  • After each task’s local training rounds, the global server aggregates updated key-prompt pairs and classifiers and stores them for continual learning.The stored parameters are redefined as the task’s key-prompt pair and classifier.Source: §4.2.2 · p. 7, §4.3 · p. 8

4.2.2 How to Explore Positive Inter-Task Relationships

  • A soft orthogonality constraint on local key and prompt gradients balances new-task learning against forgetting; its parameter ϕ controls the degree of orthogonality.At ϕ = 0, updates are fully within the orthogonal feature space, while ϕ = 1 uses the original gradients.Source: §4.2.2 · pp. 7–8

4.3 Optimization

  • Local clients detect new-task arrivals, while the proxy server pools class prototypes and uses radius-adaptive DBSCAN to estimate the task’s class count.The proxy also constructs a feature space from the pooled prototypes and sends it to local clients.Source: §4.3 · p. 8
  • Local clients compute prompt gradients for old tasks, and the global server uses the resulting information to determine the new task’s initial key-prompt pair.The proxy constructs the feature space from pooled class prototypes and shares it with the global server and local clients.Source: §4.3 · p. 8

5 EXPERIMENTS

Experiments evaluate TRPE in rehearsal-free federated continual learning across class- and domain-incremental benchmarks. The setup compares continual-learning methods and backbones using cumulative and final average accuracy.

5.1 Implementation Details

  • The evaluation spans class-incremental CIFAR-100, CUB-200, and ImageNet-R tasks, plus domain-incremental Office and Digit tasks.The CIFAR-100 comparison table covers different class-incremental settings.Source: §5.1 · p. 8, Table 1 · p. 9
  • The rehearsal-free setup adds 10 randomly selected new local clients from the second task onward and allocates no memory for storing earlier-task training images.The first incremental task starts with 30 local clients; each global round randomly selects 10 clients for local training.Source: §5.1 · pp. 8–9
  • Comparisons include representative federated continual-learning and continual-learning methods combined with federated learning, across Sup-21K, DINO-1K, and iBOT-1K backbones.All compared methods use the out-of-distribution detector and radius-adaptive DBSCAN for task discovery, so the evaluation focuses on continual-learning performance.Source: §5.1 · p. 9
  • Performance is assessed using cumulative average accuracy (CAA) and final average accuracy (FAA).FAA averages accuracy over all learned classes at the last task; CAA averages the accuracies of learned tasks.Source: §5.1 · p. 9

5.2 Performance Comparison

TRPE outperforms rehearsal-based methods, federated continual-learning baselines, and existing FCL approaches across the reported settings. Ablations attribute gains to the contrastive-learning, task-relevance, prompt-determination, and soft-orthogonality modules.

5.2 Performance Comparison

  • TRPE surpasses rehearsal-based methods by 5.50%∼43.51% in FAA and 2.48%∼39.65% in CAA across class-incremental settings, and outperforms other compared methods.The comparisons span five benchmark datasets and include federated continual-learning and prompt-based baselines.Source: §5.2 · pp. 9–10

5.3 Ablation Study

  • Adding CT to Base increases FAA by 3.19%∼20.28% and CAA by 1.40%∼19.52%.CT explores discriminative latent features for constructing feature spaces used in the task-relevance measure.Source: §5.3 · p. 10
  • The ablations report that every proposed module contributes to both class-incremental and domain-incremental FCL settings.Base excludes CT, TR, DP, and SO and is optimized with L_t = L_CE + γ_2L_MT.Source: §5.3 · p. 10
  • The TR module improves FAA by 0.54%∼5.65% and CAA by 0.59%∼18.29%.It evaluates global task relevance to determine key-prompt pairs for new tasks.Source: §5.3 · p. 10
  • The DP and SO modules improve FAA and CAA; DP incorporates prompts from previously learned tasks with higher semantic correlations to the new task.The soft orthogonality constraint facilitates learning new tasks while addressing forgetting of old tasks.Source: §5.3 · p. 11

5.4 Analysis of Task Relevance Measure

The analysis tests alternative relevance measures and model variants. It concludes that the proposed measure with an elastic threshold evaluates global semantic relationships and supports learning new tasks while alleviating forgetting.

5.4 Analysis of Task Relevance Measure

  • The analysis compares the proposed relevance measure with entropy-based information gain, Fisher information, and a dual task correlation matrix.The dual task correlation matrix alternative is adopted from Powder.Source: §5.4 · pp. 11–12
  • The proposed task-relevance measure with an elastic threshold is reported to evaluate global semantic relationships between old and new tasks and alleviate forgetting while facilitating new-task learning.The analysis compares variants that select prompts randomly, omit task relevance, or omit the elastic threshold.Source: §5.4 · pp. 12–13
  • The CIFAR-100 relevance-measure analyses cover T = 5 and T = 10 settings.The supplied table captions identify the dataset and task counts, but do not provide cell values.Source: Table 9 · p. 13, Table 10 · p. 13

5.5 Analysis of Asynchronous FCL

TRPE is reported to perform best across the evaluated asynchronous FCL settings. The authors report robustness to client-task asynchrony and continued superiority as task count increases.

5.5 Analysis of Asynchronous FCL

  • TRPE achieves the best performance across the evaluated asynchronous settings and continues to outperform alternatives as the number of tasks increases.The comparison is reported for CIFAR-100 with Sup-21K as the backbone.Source: §5.5 · p. 13, Table 11 · p. 13
  • The asynchronous evaluation shifts 30% of local clients to the next task after every three global rounds within each task.Selected clients determine a key-prompt pair for the new task before joining global aggregation with the remaining clients.Source: §5.5 · p. 13
  • The reported asynchronous results support leveraging task relevance to exploit positive inter-task relationships when learning new tasks.Source: §5.5 · p. 13

5.6 Analysis of Radius-Adaptive DBSCAN

The radius-adaptive DBSCAN groups global class prototypes into clusters to estimate the number of classes in a task. The reported examples indicate it handles noisy prototypes and outliers associated with heterogeneous client data distributions.

5.6 Analysis of Radius-Adaptive DBSCAN

  • The reported CIFAR-100 and CUB-200 examples show global class prototypes grouped into different clusters by radius-adaptive DBSCAN.The examples use randomly sampled incremental tasks.Source: §5.6 · p. 13
  • The clustering is reported to handle noisy prototypes and outliers from heterogeneous client data distributions through an adaptive radius.Source: §5.6 · p. 13
  • The analysis examines radius-adaptive DBSCAN for estimating the quantity of classes in an incremental task.Source: §5.6 · p. 13

5.7 Analysis of Computation Complexity

TRPE uses the fewest trainable parameters and least memory among the compared methods, with comparable communication costs and per-batch training and test times.

5.7 Analysis of Computation Complexity

  • TRPE has the fewest trainable parameters and lowest memory storage among the compared methods, with comparable communication costs and per-batch training and test times.The comparison is on CIFAR-100 with Sup-21K and T = 5.Source: §5.7 · p. 14, Table 12 · p. 14
  • Because Q and Lp are small relative to D, SVD on F* can be performed efficiently on a single NVIDIA 4090 GPU.The authors report that TRPE therefore has computational complexity comparable to other state-of-the-art comparison methods.Source: §5.7 · p. 14
  • TRPE achieves substantial performance improvements over the compared methods while retaining comparable computational complexity.The performance comparison is referenced in Tables 1–5.Source: §5.7 · p. 14

5.8 Analysis of Heterogeneous Data Distributions

TRPE performance worsens as client data heterogeneity increases, while its task-boundary detector remains highly accurate across tested heterogeneous settings, with an exception at the earliest boundary under extreme heterogeneity.

5.8 Analysis of Heterogeneous Data Distributions

  • Under greater client data heterogeneity, TRPE performs worse on CIFAR-100.The study distributes all images from selected class subsets of the current task across clients.Source: §5.8 · p. 14, Table 13 · p. 14

5.9 Analysis of Out-Of-Distribution Detector

  • The detector achieves 100% task-boundary accuracy for t ∈ {3, 4, 5} even when clients receive only 5% or 10% of current-task classes.At t = 2, accuracy falls below 100% under those highly heterogeneous settings.Source: §5.9 · p. 14, Table 14 · p. 14
  • At t = 2, extreme heterogeneity can leave clients with only one or two first-task classes, limiting discriminative representations and making the task-entropy difference too small to exceed δe = 0.9.The authors link this to missed task-boundary detections.Source: §5.9 · p. 14
  • Experiments verify task-boundary detection under both synchronous and asynchronous FCL settings.The asynchronous protocol transitions 30% of clients to the next task every three global communication rounds; detection accuracy is averaged over 100 runs at δe = 0.9.Source: §5.9 · pp. 14–15

5.10 Analysis of Hyperparameters

The detector's hyperparameter analyses support δe = 0.9, while TRPE is robust across tested prompt-reuse settings and all incremental tasks converge stably by five global rounds.

5.10 Analysis of Hyperparameters

  • Across the analyzed backbones and T settings, adjacent-task entropy increases exceed 0.9, while within-task entropy differences are generally below 0.9, supporting δe = 0.9 for task detection.CAA remains robust across nω ∈ {0, 1, 2, 3, 4, 5}, with best performance at nω = 1.Source: §5.10 · p. 15

5.11 Convergence Analysis

  • For TRPE, task-wise convergence is visualized on CIFAR-100 and CUB-200 with DINO-1K [5] and Sup-21K [45] backbones; all incremental tasks can effectively converge to a stable value at global round R = 5.Source: §5.11 · p. 15

6 CONCLUSION

TRPE addresses unknown task arrivals and class counts in federated continual learning while using global task relevance to select prompts and balance new-task learning against forgetting. Experiments across datasets and backbones support its effectiveness in class- and domain-incremental settings.

6 CONCLUSION

  • An out-of-distribution detector and radius-adaptive DBSCAN identify new-task arrivals and estimate new-task class counts without prior knowledge.Source: §6 · p. 15
  • TRPE measures task relevance through affinity between the new-task gradient and its component orthogonal to the old-task feature space.The global relevance guides new-task key-prompt initialization and selection of highly related old tasks whose prompts are concatenated for training.Source: §6 · p. 15
  • A soft orthogonality constraint on local key and prompt gradients balances learning new tasks against forgetting old ones.Source: §6 · p. 15
  • Experiments across diverse datasets and backbones demonstrate TRPE's effectiveness in class-incremental and domain-incremental federated continual learning.Future work will target reducing communication and computational costs, particularly under severe federated heterogeneity.Source: §6 · p. 15

Related papers

Loading 2610.34461v1…