PaperlayerView exact revision on arXiv

Source-linked AI summary

Scalable and Order-robust Continual Learning with Additive Parameter Decomposition

Jaehong Yoon, Saehoon Kim, Eunho Yang, Sung Ju Hwang

arXiv:1902.09432v3cs.LGstat.ML

TL;DR

Continual learning needs methods that prevent forgetting and remain scalable and order-robust as tasks accumulate, especially where task-order disparities affect fairness. The paper proposes APD-Net, which combines shared and sparse task-adaptive parameters with hierarchical consolidation, and reports superior benchmark performance across accuracy, efficiency, scalability, and order-robustness. Its stated setup assumes a sequence of tasks arriving in random order.

  • Problem

    Conventional continual-learning methods have not been verified for effective forgetting prevention, memory and computation efficiency at large task counts, or robustness to task-arrival order.

  • Method

    APD-Net decomposes network parameters into task-shared and sparse task-adaptive components, with hierarchical consolidation clustering task-adaptive parameters into shared parameters.

  • Results

    Across multiple benchmark datasets, APD-Net is reported to outperform recent continual-learning methods in accuracy, efficiency, scalability, and order-robustness with minimal network-capacity increase.

  • Takeaways & Limitations

    The decomposition preserves earlier task-adaptive parameters, avoids topology expansion, and supports scalable, order-robust continual learning with hierarchical knowledge consolidation.

  • Takeaways & Limitations

    The framework assumes a sequence of tasks arriving to a deep network in random order.

Abstract

from arXiv · show

While recent continual learning methods largely alleviate the catastrophic problem on toy-sized datasets, some issues remain to be tackled to apply them to real-world problem domains. First, a continual learning model should effectively handle catastrophic forgetting and be efficient to train even with a large number of tasks. Secondly, it needs to tackle the problem of order-sensitivity, where the performance of the tasks largely varies based on the order of the task arrival sequence, as it may cause serious problems where fairness plays a critical role (e.g. medical diagnosis). To tackle these practical challenges, we propose a novel continual learning method that is scalable as well as order-robust, which instead of learning a completely shared set of weights, represents the parameters for each task as a sum of task-shared and sparse task-adaptive parameters. With our Additive Parameter Decomposition (APD), the task-adaptive parameters for earlier tasks remain mostly unaffected, where we update them only to reflect the changes made to the task-shared parameters. This decomposition of parameters effectively prevents catastrophic forgetting and order-sensitivity, while being computation- and memory-efficient. Further, we can achieve even better scalability with APD using hierarchical knowledge consolidation, which clusters the task-adaptive parameters to obtain hierarchically shared parameters. We validate our network with APD, APD-Net, on multiple benchmark datasets against state-of-the-art continual learning methods, which it largely outperforms in accuracy, scalability, and order-robustness.

1 INTRODUCTION

Continual learning must address catastrophic forgetting, scalability to many tasks, and task-order sensitivity in addition to learning sequentially. APD-Net decomposes parameters into shared and sparse task-adaptive components, aiming to improve these properties while retaining efficient training and capacity growth.

  • Continual learning incrementally updates a model across tasks but risks catastrophic forgetting as new-task training shifts previously learned representations.
  • Large task sequences expose scalability limits in conventional methods, including unverified forgetting prevention and increased memory and computation requirements.
  • Task-order sensitivity causes performance discrepancies across arrival sequences and can threaten fairness when task-level outcomes matter, such as in disease diagnosis.
  • APD-Net represents each layer’s parameters with task-shared and sparse task-adaptive components, using small masks while learning only task-specific residual differences.
  • Hierarchical knowledge consolidation clusters task-adaptive parameters into hierarchically shared parameters, while experiments report superior accuracy, efficiency, scalability, and order-robustness against recent methods.
  • Earlier task-adaptive parameters remain unaffected by later-task learning, while shared parameters capture generic knowledge, reducing forgetting and helping stabilize performance across task orders.

2 RELATED WORK

Related continual-learning work has primarily addressed catastrophic forgetting through regularization, generative replay, parameter constraints, and masking. Dynamic network expansion offers another response to unlimited task sequences by increasing model capacity for new tasks.

  • Continual Learning: Recent continual-learning methods primarily prevent catastrophic forgetting by limiting shifts in learned representations during retraining for new tasks.
  • Continual Learning: Representative approaches use representation regularization, deep generative models, Fisher-weighted parameter regularization, moment matching, or binary masking.
  • Dynamic Network Expansion: Dynamic network expansion addresses potentially unlimited task sequences by adding capacity for new tasks, following earlier expansion-based approaches.

3 CONTINUAL LEARNING WITH ADDITIVE PARAMETER DECOMPOSITION

The paper decomposes each task's network parameters into task-shared and sparse task-adaptive components, then updates prior task adaptations to compensate for shared-parameter changes. Hierarchical consolidation further groups related task adaptations into locally shared parameters to reduce capacity growth.

  • Order Robust Continual Learning: The training objective combines current-task loss with sparsity and shared-parameter transfer regularization, balancing efficiency against catastrophic forgetting.The model is trained incrementally because previous task datasets are assumed inaccessible after their training steps.
  • Additive Parameter Decomposition: APD represents task t parameters as θt = σ ⊗ Mt + τt, combining masked shared parameters σ with task-adaptive parameters τt.The mask Mt guides shared-parameter use toward task-relevant components, while τt captures task-specific residual knowledge.
  • Additive Parameter Decomposition: Sparse τt stores task-specific knowledge that cannot be explained by transformed shared knowledge, reducing the parameters learned for each arriving task.An ℓ1 penalty encourages sparsity in the task-adaptive parameters.
  • Order Robust Continual Learning: Later-task learning preserves earlier task-adaptive parameters while changing shared parameters primarily through generic knowledge, limiting forgetting and order sensitivity.The method also regularizes shared-parameter changes and retroactively updates prior task adaptations to preserve their original solutions.
  • Hierarchical Knowledge Consolidation: Hierarchical knowledge consolidation clusters task-adaptive parameters and decomposes each group into locally shared and more-sparse task-adaptive components.Every s tasks, K-means clustering updates local shared parameters and increases the number of centroids from K to K + k as task variance grows.

4 EXPERIMENT

APD-Net is evaluated across benchmark datasets for accuracy, efficiency, catastrophic forgetting, order-robustness, selective forgetting, and scalability. The experiments report strong accuracy with minimal capacity increase, marginal training-time overhead, low order disparity, and near-single-task performance on 100 tasks.

  • Task-average performance: APD-Nets outperform baselines in task-average accuracy while using lower capacity than expansion-based methods.The reported evaluation averages results over three runs and five task-order sequences on CIFAR-100 datasets.
  • Efficiency: APD(1) and APD(2) require training time comparable to the base model, with only a marginal increase.Unlike several expansion-based approaches, APD-Net can be trained in a single training step despite updating earlier task-adaptive parameters.
  • Order fairness in continual learning: APD-Nets obtain significantly lower MOPD and AOPD than baseline models, showing the least disparity across task orders.The comparison uses five random task sequences, while Figure 4 visualizes per-task accuracy for three different orders.
  • Preventing catastrophic forgetting: APD-Nets show no sign of catastrophic forgetting as new tasks arrive, while APD(2) improves task 6 by 0.40%p.The analysis examines tasks 1, 6, and 11 during learning on the 20-task CIFAR-100 Superclass dataset.
  • Selective task forgetting: Removing Task 3 or Task 5 parameters causes no performance degeneration on non-target tasks.This experiment evaluates selective task forgetting on Tasks 1–5 using APD(1).
  • Scalability to large number of tasks: On 100-task Omniglot-Rotation, APD nearly matches single-task learning while using far fewer network parameters.The paper reports that APD outperforms other continual-learning models by a large amount; STL uses 100 times more network parameters.
  • Qualitative analysis: Parameter visualizations show that APD-Fixed largely alleviates semantic drift, while APD-Net additionally uses retroactive updates.L2-Transfer parameters drift in a new direction during sequential training, whereas later-task updates primarily affect APD shared parameters.

5 CONCLUSION

The paper concludes that Additive Parameter Decomposition separates generic and task-specific knowledge to address forgetting and order sensitivity efficiently. Experiments support accuracy gains over existing approaches with memory, computation, scalability, and order-robustness advantages.

  • APD represents task parameters as task-shared parameters plus task-adaptive parameters that capture task-specific differences.
  • Earlier tasks’ task-adaptive parameters remain intact, while relatively static shared parameters and retroactive updates address task order sensitivity.
  • APD is more efficient than expansion-based approaches because additive task-adaptive parameters do not increase the number of neurons or filters.
  • Extensive experiments showed accuracy gains over existing continual learning approaches while maintaining memory and computation efficiency, scalability, and order-robustness.

A APPENDIX

The appendix provides detailed APD experiment settings, quantitative analyses, and ablation studies.

  • The appendix describes detailed experimental settings for models using Additive Parameter Decomposition.
  • It reports experimental results with additional quantitative analysis and an ablation study for APD.
  • The appendix supplements the main evaluation with analyses intended to assess APD’s behavior and design.

A.1 EXPERIMENT SETTINGS

Experiments use dataset-specific architectures, training schedules, regularization settings, and multiple task orders, with an ablation table on CIFAR-100 Split.

  • A.1 EXPERIMENT SETTINGS: MNIST-Variation uses two-layer feedforward networks with 312 and 128 neurons, 50 training epochs, and dataset-specific APD λ1 values.
  • A.1 EXPERIMENT SETTINGS: CIFAR-100 Split and Superclass use LeNet with 20-50-800-500 neurons and 20 training epochs, with hierarchical consolidation settings K=2 per 5 tasks.
  • A.1 EXPERIMENT SETTINGS: Omniglot uses LeNet with 10-20-500-300 neurons by default, while a larger 64-128-2500-1500 network evaluates EWC with greater capacity.
  • A.1 EXPERIMENT SETTINGS: Order robustness is estimated using 5 different task orders across experiments, with random orders selected for MNIST-Variation and CIFAR-100 datasets.
  • A.1 EXPERIMENT SETTINGS: Table A.3 reports APD(1) ablation results averaged over five task orders on CIFAR-100 Split.

A.2 ARCHITECTURAL CHOICES FOR ADDITIVE PARAMETER DECOMPOSITION

Ablations examine sparsity and masking, while appendix comparisons evaluate APD against GEM variants and HAT under specified continual-learning settings.

  • A.2 ARCHITECTURAL CHOICES FOR ADDITIVE PARAMETER DECOMPOSITION: Removing sparsity produces substantially higher accuracy but requires impractically large capacity.
  • A.2 ARCHITECTURAL CHOICES FOR ADDITIVE PARAMETER DECOMPOSITION: Table A.4 compares APD with GEM variants on Permuted-MNIST using the A-GEM experimental settings and episodic memory of 256.
  • A.2 ARCHITECTURAL CHOICES FOR ADDITIVE PARAMETER DECOMPOSITION: Table A.5 compares APD-Net with HAT on eight heterogeneous datasets across five randomly generated task-order sequences.

A.3 COMPARISON WITH OTHER CONTINUAL LEARNING METHODS

The appendix compares APD and APD-Net with GEM-based approaches and HAT across heterogeneous and CIFAR-100 continual-learning experiments. These comparisons address capacity limitations, knowledge transfer, and task-order sensitivity.

  • GEM comparison: APD is compared with GEM-based approaches using a two-layer perceptron with 256 neurons per layer.GEM variants use episodic memories of previous-task data to compute gradients during later-task training.
  • Heterogeneous datasets: APD-Net is evaluated against HAT on eight heterogeneous datasets using a modified AlexNet backbone.The sequence includes CIFAR-10, CIFAR-100, FaceScrub, MNIST, NotMNIST, FashionMNIST, SVHN, and TrafficSign.
  • Method comparison: GEM variants and HAT cannot expand network capacity and/or perform unidirectional knowledge transfer, leading to capacity limitation and order-sensitivity.The passage characterizes these as strong continual-learning approaches despite those constraints.
  • APD mechanism: APD adaptively increases network capacity through task-adaptive parameters that learn task-specific features absent from task-shared parameters.This parameterization supports richer representations than fixed-capacity continual-learning methods.
  • Evaluation protocol: CIFAR-100 Split and CIFAR-100 Superclass results report mean accuracies over three runs, five task-order sequences, and standard deviations.The experiments use random splits.
  • Order robustness: Per-task accuracy across five differently ordered CIFAR-100 Split sequences reveals task-order sensitivity when performance disparities are large.The figure frames order variation as relevant to fairness in continual learning.
Loading 1902.09432v3…