PaperlayerView exact revision on arXiv

Source-linked AI summary

One Adapter, Many Tasks: Task-Conditioned Feature Transformations for Continual Learning

Yunxiang Fu, Meng Lou, Yizhou Yu

arXiv:2608.31096v1cs.CVcs.LG

TL;DR

Class-incremental learning needs to acquire new classes without earlier data while retaining recognition of all seen classes, but existing adapters and LoRA merging face efficiency or interference limitations. FACET uses one shared adapter with dynamic task-conditioned transformations and replay-free feature consistency. It achieves strong accuracy across 10–200-task benchmarks while reducing task-specific costs.

  • Problem

    Existing task-specific adapters are parameter- and computation-inefficient, while static merged LoRA weights can produce representation interference during inference.

  • Method

    FACET uses a single shared adapter with dynamic task-conditioned transformations and a replay-free consistency loss aligned to a frozen adapter snapshot.

  • Results

    FACET achieves the highest final accuracy on four benchmarks spanning 10–200 tasks, including 79.97% on ImageNet-R with 20 tasks and 3.45% and 6.81% gains over MIN and TUNA on OmniBenchmark-1K with 200 tasks.

  • Takeaways & Limitations

    A single adapter can provide task-dependent, more discriminative features while supporting efficient and scalable continual learning across long and short task sequences.

  • Takeaways & Limitations

    The method assumes exemplar-free CIL with a frozen pretrained vision backbone, disjoint task labels, and known training task boundaries; its replay-free objective may be weaker for highly divergent task distributions.

Abstract

from arXiv · show

Class-incremental learning (CIL) requires a model to incrementally learn tasks that contain new classes without accessing earlier training data while preserving the ability to recognize all seen classes. Recently, pretrained-model-based approaches have become prevalent by adapting a frozen backbone with additional lightweight trainable modules. Existing methods, however, exhibit limitations: task-specific adapters learn explicit per-task representations but are parameter- and computation-inefficient, while LoRA-based merging methods combine per-task LoRA parameters into a single model whose static aggregated weights cause representation interference during inference. To address these problems, we present \textbf{FACET}: task-conditioned \textbf{F}e\textbf{A}ture transformation with \textbf{C}ondition\textbf{E}d feature consis\textbf{T}ency, achieving excellent parameter efficiency while producing highly discriminative features during inference. When continually trained on a task sequence, FACET learns a single shared adapter that employs a dynamic task-conditioned feature transformation, shaping the overall feature distribution of the adapter into a mixture of overlap-reduced task-specific components. On the other hand, we propose an efficient replay-free task-conditioned feature consistency loss, aiming to mitigate catastrophic forgetting of the learned mixture distribution in the adapter's feature space. Even when maintaining only a single adapter, FACET demonstrates robust scalability. On both very long task sequences (e.g., 200 tasks) and standard short task sequences (e.g., 20 tasks), our method achieves superior performance while using significantly fewer trainable parameters and GFLOPs. The code will be made open source upon acceptance.

1 Introduction

Class-incremental learning must add new classes while retaining earlier knowledge, but existing pretrained-model adaptations trade off efficiency against representation interference. FACET addresses this with one task-conditioned adapter and replay-free consistency, achieving strong results across task-sequence lengths.

  • Motivation: Class-incremental learning requires recognizing new and previously seen classes from sequential data without forgetting earlier tasks.
  • Representation quality: FACET achieves higher class-averaged silhouette scores than SD-LoRA and TUNA, indicating more class-discriminative features across adapter blocks.
  • Limitations of existing methods: Task-specific adapters use explicit per-task representations but are parameter- and computation-inefficient, while merged LoRA weights can cause representation interference.
  • FACET: FACET uses a single continually optimized adapter with dynamic task-conditioned transformations that improve task-specific feature separability.
  • Results: FACET achieves the highest final accuracy on ImageNet-R, ImageNet-A, CIFAR-100, and ObjectNet across benchmarks with 10 to 200 tasks.On ImageNet-R with 20 tasks, accuracy improves from 75.26% to 79.97% versus SD-LoRA; on OmniBenchmark-1K with 200 tasks, it improves over MIN and TUNA by 3.45% and 6.81%.
  • FACET: A task-conditioned consistency objective preserves earlier conditional feature distributions by aligning the latest adapter with a frozen snapshot.

2 Related Work

Related work addresses continual-learning forgetting through regularization, replay, architectural expansion, and parameter isolation. Pretrained-model CIL methods increasingly use task-specific prompts or adapters, while parameter-efficient alternatives merge task modules into a static model.

  • Continual learning: Earlier continual-learning methods address forgetting through regularization, knowledge distillation, replay, parameter isolation, and architectural expansion.
  • Pretrained-model CIL: Pretrained-model CIL methods attach lightweight task-specific parameters, including retrieved prompt pools and dedicated adapters.
  • Parameter-efficient CIL: Parameter-efficient methods such as InfLoRA and SD-LoRA merge separately trained task modules into a single static inference model.

3 Method

FACET uses one shared adapter whose output is dynamically transformed according to task-related context, producing task-dependent representations without task-specific adapter weights. A replay-free consistency objective preserves earlier conditional feature distributions while the method remains computationally efficient.

  • 3.1 Problem Formulation: CIL trains sequentially on disjoint task classes, evaluates on all seen classes, and follows an exemplar-free setting without historical data.Only the latest task data are accessible during training, while the frozen backbone and expandable classifier support recognition across seen classes.
  • 3.2 Single Adapter with Dynamic Task-Conditioned Transformations: FACET equips a frozen ViT with one shared adapter and conditions its output on a context vector reflecting task-related information.The context vector is used to obtain task-dependent features while adapter weights remain shared across tasks.
  • 3.2 Single Adapter with Dynamic Task-Conditioned Transformations: The adapter computes a common latent representation, then dynamically transforms it with a context-generated matrix before shared up-projection to the ViT feature space.Different task contexts produce different transformations of the same latent representation; since up-projection is linear, this suffices to make final adapter outputs task-dependent.
  • 3.2 Single Adapter with Dynamic Task-Conditioned Transformations: The task-conditioned matrix combines independent feature scaling with low-rank cross-dimension mixing, mapped from the context vector to preserve parameter efficiency.Scaling resizes latent dimensions independently, while mixing redistributes activations across dimensions to create different task-specific linear combinations.
  • 3.2 Single Adapter with Dynamic Task-Conditioned Transformations: Task-conditioned transformations reduce overlap among task-specific mixture components, whereas explicit separation losses can reduce adapter capacity and produce suboptimal ablation results.The overall adapter feature distribution is treated as a mixture of task-specific components whose locations and shapes can be altered by conditioning.
  • 3.3 Task-Conditioned Feature Consistency: The consistency objective copies the previous adapter, samples an earlier task context, and compares conditioned [CLS] features on current-task samples to limit distribution drift without historical data.Using the same input under earlier-task conditioning measures drift from the learned feature distribution; experiments report preserved historical distributions and substantially mitigated forgetting.

4 Experiments

FACET is evaluated across standard and long-sequence class-incremental learning benchmarks, with ablations examining its components and design choices. The experiments report strong accuracy alongside substantial efficiency gains and sustained performance as task counts increase.

  • Experimental Setup: FACET is evaluated on CIFAR100, ImageNet-R, ImageNet-A, ObjectNet, and OmniBenchmark-1K, covering sequences of up to 200 tasks.The evaluation uses standard and distribution-shifted visual domains with short, medium-length, and long task sequences.
  • Experimental Setup: The experiments compare FACET against prompt-based, adapter-based, classifier-based, and LoRA-based weight-merging methods.The baselines include both task-specific-parameter methods and single-model merging approaches.
  • Main Results: FACET achieves the highest final accuracy on all four standard datasets with 10 to 20 tasks.On ImageNet-R with 20 tasks, final accuracy rises from 78.08% with MIN to 79.97%; on ImageNet-A, FACET reaches 64.87% final accuracy and improves over the strongest LoRA-based method by 5.09%.
  • Efficiency: At 200 tasks, FACET achieves a 156× inference-latency speedup and uses 0.57% of MoS’s GFLOPs, while requiring 1.25% of MIN’s cumulative trainable parameters.Table 3 reports median latency at batch size 32 and compares trainable parameters, computation, and GFLOPs.
  • Ablation Studies: Ablations show that task-conditioned transformation raises final accuracy from 48.75% to 60.02%, while adding task-conditioned consistency raises it to 78.23%.Non-conditioned consistency reaches 65.39%, indicating that the conditioned transformation and consistency components contribute differently.

5 Limitations

FACET assumes exemplar-free CIL with a frozen pretrained vision backbone, disjoint task labels, and known training task boundaries. Its replay-free consistency objective may not match the full old-data distribution, while single-adapter capacity may saturate on longer or highly heterogeneous sequences.

  • FACET assumes exemplar-free CIL with a frozen pretrained vision backbone, disjoint task label sets, and known task boundaries during training.
  • Task-specific auxiliary-classifier costs remain small but do not fully disappear, despite substantial reductions for 200-task sequences.
  • A single adapter may saturate on longer task sequences or highly heterogeneous tasks, making extension to thousands of tasks non-trivial.
  • Matching a frozen snapshot on current-task inputs avoids storing old data but is not equivalent to matching the full old-data distribution.
  • The replay-free objective may be weaker when new-task data differ substantially from earlier tasks.

6 Conclusion

FACET is a single-adapter approach for pretrained-model-based CIL that combines task-conditioned latent transformation with preservation of earlier task-specific feature distributions. It is evaluated across five image datasets spanning standard, artistic, adversarial, object-centric, and large-scale settings.

  • FACET uses one adapter shared across tasks, dynamically transforming adapter representations according to task context while preserving previously learned task-specific feature distributions.
  • The evaluation covers five datasets: ImageNet-R, ImageNet-A, CIFAR100, ObjectNet, and OmniBenchmark-1K.
  • ImageNet-R contains 200 classes and 30000 images depicting ImageNet-compatible categories in artistic and non-standard visual styles.
  • ImageNet-A contains 200 classes and 7475 naturally adversarial images with ImageNet-compatible categories.
  • CIFAR100 contains 100 classes and 60000 images, while ObjectNet contributes 200 classes and 33137 images with varied backgrounds, rotations, and viewpoints.
  • OmniBenchmark-1K contains 188569 images across 1000 classes with data non-overlapping with ImageNet-1k.

A.2 Implementation Details

The experiments use PILOT with pretrained ViT-B/16 backbones and insert one task-conditioned adapter per transformer block. Results across shuffled task orders and two backbones show consistent superiority over strong baselines.

  • Main experiments use ViT-B/16 pretrained on ImageNet-21k, with one shared task-conditioned adapter inserted in each transformer block.Each adapter has two 384-dimensional hidden layers, and context-generation projections are zero-initialized at the first training step.
  • Tables 8–9 compare ImageNet-R (Inc10) performance using ViT-B/16-IN21k and ViT-B/16-IN1k across five random seeds.The reported metric is Ā / AT.
  • FACET outperforms strong baselines across all shuffled task orders on both pretrained backbones.Task order is randomized with five seeds, indicating the advantage is not attributed to one particular task order.

B.2 Detailed Ablations on Single Adapter Design

Ablations support stable initialization, sparse context design, and task-conditioned consistency choices, while showing that performance is robust across adapter capacities and Kmax values. Adding context noise improves robustness to imperfect task identity prediction.

  • 79.97% AT results from adding noise to the context vector, versus 78.80% with zero initialization and 76.68% with random initialization.
  • FACET remains stable across bottleneck dimensions and adapter depths, so its gains are not simply due to adding more adapter parameters.
  • Smaller Kmax values perform better for ImageNet-R (Inc10) with 20 tasks, while Kmax = 200 is selected as a sufficiently large upper bound.

B.3 Impact of the Coefficient for Task-Dependent Feature Consistency Loss

The task-conditioned feature consistency coefficient balances preserving task-specific feature distributions against learning new tasks.

  • λcon = 2.0 achieves the highest final accuracy and optimally balances mitigating forgetting with learning the new task.

C Analysis of Task Prediction Accuracy

FACET’s task-context predictions substantially exceed random guessing, while incorrect contexts cause only limited accuracy degradation. Ground-truth contexts can further improve performance.

  • Random and forced-wrong context results are measured without retraining on the same checkpoint and averaged over five tries with sample standard deviations.
  • 5.3 to 9.8 times above random guesses is the task prediction accuracy achieved by FACET.
  • 4.48 points is the maximum final-accuracy reduction from using an incorrect context, from 79.97 to 75.49 on ImageNet-R with 20 tasks.
  • 79.97 to 81.62 is the final-accuracy improvement on ImageNet-R with 20 tasks when ground-truth task context is provided during inference.

D Additional Analysis of Task-Conditioned Feature Consistency

The conditioned feature consistency loss preserves historical task features during continual training, substantially reducing feature drift after many subsequent tasks. Feature drift is strongly associated with old-task classification forgetting.

  • Figure 3 reports mean distances as solid lines and standard deviations as light shaded regions.
  • 0.0588 cosine distance and 0.0866 MSE remain after 99 subsequent tasks with the consistency loss, versus 0.249 and 0.427 without it.Without the loss, cosine distance increases by 323% and MSE by 393%.
  • The analysis compares first-task CLS features after learning t ∈{2, ..., 100} tasks with their original features using the same first-task context.CLS features from the last transformer block are used because they are passed to the classifier.
  • Cosine distance captures directional feature drift, whereas MSE captures overall displacement in feature space.
  • 0.752 and 0.711 are the Pearson correlations of cosine distance and MSE, respectively, with task-1 forgetting across six λcon settings.Greater feature drift is strongly associated with greater old-task classification forgetting.

E Justification of Using Current-Task Data for Conditioned Feature Consistency

Current-task probes can constrain changes to a shared adapter under earlier task contexts when current and old inputs respond to overlapping adapter directions. Empirical measurements support this coverage interpretation without requiring semantic similarity between the inputs.

  • The consistency loss uses current-task data Dt to preserve old feature distributions rather than replaying old-task images.Current-task images serve as probes for whether new-task training changes the shared adapter under previous contexts.
  • Using identical current inputs and sampled old contexts in frozen and updated adapters isolates the effect of adapter updates under previous contexts.Minimizing the resulting feature difference constrains new-task training from changing the adapter’s behavior under those contexts.
  • Because all tasks use the same low-dimensional adapter and the backbone is frozen, weight changes affecting old features can also alter current probes and incur consistency penalties.This does not require identical images, labels, or feature distributions across tasks.
  • Current-task images activate most adapter directions important for first-task features when evaluated under the old-task context.The experiment concerns shared adapter directions, not semantic similarity between old and current images.
  • 0.869±0.049 average similarity and 99.58% directional satisfaction support overlap between current-task and old-task adapter directions after 20 tasks.The comparison uses current images from the last task and old tasks 0, 4, 8, 12, and 16 under each old task’s context.
Loading 2608.31096v1…