Source-linked AI summary
Reinforced Continual Learning
Ju Xu, Zhanxing Zhu
TL;DR
Continual learning must acquire sequential tasks without forgetting prior knowledge. RCL uses reinforcement learning to search for task-specific network architectures, and experiments on MNIST variants and CIFAR-100 report stronger accuracy and lower complexity than existing alternatives.
Problem
Continual learning seeks to learn new sequential tasks without forgetting previously acquired knowledge.
Method
RCL uses reinforcement learning to search for the best neural architecture for each coming task by adaptively selecting added nodes or filters.
Results
RCL outperforms existing continual-learning alternatives on sequential MNIST variants and CIFAR-100 tasks in prediction accuracy and model complexity.
Takeaways & Limitations
RCL prevents forgetting while fitting new tasks and can achieve better performance with substantially fewer added parameters than expandable-network baselines.
Takeaways & Limitations
RCL requires many reinforcement-learning trials, resulting in longer training time than other methods; reducing this cost remains future work.
Abstract
from arXiv · showhide
Most artificial intelligence models have limiting ability to solve new tasks faster, without forgetting previously acquired knowledge. The recently emerging paradigm of continual learning aims to solve this issue, in which the model learns various tasks in a sequential fashion. In this work, a novel approach for continual learning is proposed, which searches for the best neural architecture for each coming task via sophisticatedly designed reinforcement learning strategies. We name it as Reinforced Continual Learning. Our method not only has good performance on preventing catastrophic forgetting but also fits new tasks well. The experiments on sequential classification tasks for variants of MNIST and CIFAR-100 datasets demonstrate that the proposed approach outperforms existing continual learning alternatives for deep networks.
1 Introduction
Continual learning seeks to learn sequential tasks while retaining earlier capabilities and leveraging prior knowledge. RCL adaptively expands neural networks with reinforcement learning to balance new-task performance, forgetting, and model complexity.
- Continual learning aims to learn consecutive tasks without forgetting previous tasks while leveraging earlier knowledge for better performance or faster training.
- Existing deep-learning strategies either preserve a fixed architecture with regularization or episodic memory, or dynamically expand the network for new tasks.
- Dynamic expansion can create redundant, expensive networks, while DEN introduces multiple sensitive regularization and thresholding hyperparameters.
- RCL poses per-layer node or filter selection as combinatorial optimization and uses an actor-critic-trained recurrent controller to choose architectural hyperparameters.
- RCL reports better performance than existing methods while adding much less units across varied sequential tasks.
2 Preliminaries of Reinforcement learning
Reinforcement learning trains an agent to select actions in an environment using rewards. Policy-gradient methods optimize parameterized stochastic policies, with discounted returns and value functions providing the learning objectives.
- Reinforcement learning learns a policy for an agent interacting with an unknown environment through states, actions, and reward signals.
- The agent seeks a policy that maximizes the expected sum of discounted rewards, where γ determines the importance of future rewards.
- A policy’s value function is its expected return, while its action-value function evaluates expected return for a state-action pair.
- Policy-gradient methods use stochastic gradient descent to optimize an objective over parameterized stochastic policies πθ(a|s).
3 Our Proposal: Reinforced Continual Learning
RCL adaptively expands a task network by using a recurrent controller to choose layer-wise additions, while preserving previous-task parameters. Reinforcement learning selects architectures using validation accuracy and network complexity, with added parameters trained for each new task.
- Framework: RCL uses a controller, value network, and task network to generate policies, estimate state values, and solve each task.The controller is implemented as an LSTM, the value network approximates state value, and the task network can be adapted to different applications.
- Adaptive expansion: For each new task, the controller chooses how many filters or nodes to add in every network layer.The choices form a fixed-length action sequence over the m layers, with each layer’s action sampled from a softmax distribution.
- Training and retention: Previous-task parameters remain fixed while only newly added parameters are trained for the incoming task.After training, newly added filters are timestamped by layer shape, and task-specific inference uses the parameters introduced for that task.
- Adaptive expansion: Exhaustively searching layer-wise additions has O(Q^m ∏_i n_i) complexity and is NP-hard for deep architectures.The controller replaces traversal of all combinatorial action combinations with autoregressive sequence generation.
- Reward and optimization: The reward combines validation accuracy with expanded-network complexity, and policy gradients train the controller because the reward is non-differentiable.The parameter α balances prediction performance against model complexity; the best-reward expanded configuration is stored for later tasks.
- Comparison: Compared with progressive networks and DEN, RCL reports more stable or better performance while adding fewer units and using fewer hyperparameters.RCL freezes prior parameters rather than selectively retraining and reduces tuning mainly to the accuracy–complexity reward balance.
4 Experiments
Experiments on sequential MNIST and CIFAR-100 tasks compare RCL with fixed-size and expandable baselines across accuracy, complexity, forgetting, and training time. RCL achieves strong accuracy with fewer parameters and prevents forgetting, while reinforcement-learning search increases training time.
- Experimental setup: Experiments use MNIST Permutations, MNIST Mix, and Incremental CIFAR-100, with ten sequential tasks evaluated against five continual-learning baselines.The baselines are SN, EWC, GEM, PGN, and DEN.
- Accuracy and complexity: Fixed-size methods have lower model complexity but much worse prediction accuracy than expandable networks such as PGN, DEN, and RCL.The comparison covers average test accuracy and model complexity across the datasets.
- Accuracy and complexity: RCL outperforms PGN and DEN on test accuracy and model complexity, reducing parameters by 42% and 53% versus PGN and DEN on incremental CIFAR-100.The reported reductions are relative to the respective expandable-network baselines.
- Accuracy and complexity: RCL matches PGN and DEN accuracy with fewer parameters and improves accuracy at comparable network sizes, indicating benefits from adaptive architectural control.Figure 3 varies hyperparameter settings to compare accuracy against model complexity for RCL, DEN, and PGN.
- Forgetting behavior: RCL and PGN show no forgetting on the first task as additional tasks are learned, whereas non-expanding methods exhibit catastrophic forgetting and DEN does not completely prevent forgetting.The paper attributes DEN’s residual forgetting to retraining previous parameters during new-task learning.
- Training time: RCL requires more training time than other methods because reinforcement learning typically needs many trials, while training-efficiency improvements remain future work.Training time is reported as wall-clock time in seconds for all compared methods.
- Reward tradeoff: Increasing α substantially reduces model complexity while gradually degrading test accuracy; when α is small, accuracy declines more slowly than parameter count.The reward coefficient α therefore controls the accuracy–complexity tradeoff.
5 Conclusion
Reinforced Continual Learning adaptively searches for neural architectures with reinforcement learning, increasing capacity when needed while preventing semantic drift. Experiments report significantly better prediction accuracy and model complexity than existing baselines, while future work targets backward transfer and training time.
- Reinforced Continual Learning searches for the best neural architecture for each incoming task using reinforcement learning.The framework increases network capacity when necessary and is designed to prevent semantic drift.
- Experiments report significantly better prediction accuracy and model complexity than existing baselines.The evaluation covers fully connected and convolutional task networks on different datasets.
- Future work will develop backward-transfer strategies so learning new tasks can improve performance on previous tasks.
- Reducing RCL training time is important for large networks with more layers.
A Experiment settings
The experiments use fixed training configurations for MNIST variants and CIFAR-100, with specified controller and value-network settings. RCL, DEN, and PGN use a search space of 30 across all layers when expanding networks.
- MNIST settings: MNIST Permutations and MNIST Mix use a three-layer 784-312-128-10 network, learning rate 0.001, batch size 32, and 15 training epochs.
- CIFAR-100 settings: CIFAR-100 experiments use LeNet as the task network with 20 training epochs.
- Network expansion: RCL, DEN, and PGN use a search space of 30 across all layers when expanding the network.
- Controller settings: The RCL controller is a two-layer LSTM with hidden size 100, while the value network is a one-layer fully connected network.
- Optimization settings: The controller and value-network learning rates are 0.001 and 0.005, respectively.
- Reward settings: The reward coefficient α is 0.0003 for MNIST Permutations, 0.0002 for MNIST Mix, and 0.001 for CIFAR-100.