Source-linked AI summary
Progressive Neural Networks
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, Raia Hadsell
TL;DR
Continual learning requires transferring knowledge across task sequences while retaining earlier capabilities, a limitation of destructive finetuning. The paper introduces progressive networks with frozen prior columns and lateral connections, evaluates them across reinforcement-learning tasks, and reports stronger transfer than common baselines while preserving prior features. It also analyzes transfer through perturbation and Fisher sensitivity measures, with transfer observed across policy layers.
Problem
Learning sequences of tasks requires transfer without catastrophic forgetting, while finetuning discards previously learned functions and leaves initialization and task similarity unresolved.
Method
Progressive networks add a new column per task, keep earlier columns pretrained and frozen, and learn lateral connections from prior features into the new column.
Results
Progressive networks provide comparable or slightly better transfer than finetuning without its destructive consequences, outperform baseline 3 across Atari transfer outcomes, and support sensitivity-based transfer analysis.
Takeaways & Limitations
The architecture accumulates experiences, remains robust to harmful features from incompatible tasks, and can exploit transfer for compatible source and target domains.
Takeaways & Limitations
Parameter count grows with the number of tasks, only a fraction of added capacity is used, and inference requires knowledge of the task label.
Abstract
from arXiv · showhide
Learning to solve complex sequences of tasks--while both leveraging transfer and avoiding catastrophic forgetting--remains a key obstacle to achieving human-level intelligence. The progressive networks approach represents a step forward in this direction: they are immune to forgetting and can leverage prior knowledge via lateral connections to previously learned features. We evaluate this architecture extensively on a wide variety of reinforcement learning tasks (Atari and 3D maze games), and show that it outperforms common baselines based on pretraining and finetuning. Using a novel sensitivity measure, we demonstrate that transfer occurs at both low-level sensory and high-level control layers of the learned policy.
1 Introduction
The paper introduces progressive networks to transfer knowledge across task sequences without catastrophic forgetting. It evaluates the architecture in complex reinforcement-learning domains and analyzes where transfer occurs.
- Architecture: Progressive networks combine previously learned features through lateral connections while retaining pretrained models throughout training.This makes prior knowledge available at each layer rather than only at initialization.
- Motivation: The architecture is designed to support transfer across sequences of tasks without destructive loss of previously learned functions.The paper contrasts this goal with finetuning, which discards the previously learned function during adaptation.
- Contribution: The combination of existing ingredients for solving complex task sequences is presented as novel.The paper identifies this combination and its use in complex sequences as a contribution.
- Evaluation: The authors extensively evaluate progressive networks and alternative transfer approaches in complex reinforcement-learning domains.The evaluation includes comparisons with finetuning within reinforcement learning.
- Analysis: A Fisher Information and perturbation-based analysis examines how and where transfer occurs across tasks.The analysis is intended to identify transfer across different learned features and network depths.
2 Progressive Networks
Progressive networks add a new trainable column for each task while preserving earlier columns and connecting new layers laterally to prior features. This design targets independent task solution, possible transfer, and avoidance of catastrophic forgetting, but parameter count grows with task number.
- Column construction: Each new task receives a separately instantiated column, while earlier columns remain frozen.The first column is trained to convergence before subsequent columns are added.
- Design goals: The architecture is designed to solve K independent tasks, accelerate learning through transfer when possible, and avoid catastrophic forgetting.These goals directly motivate the modeling decisions.
- Transfer mechanism: Lateral connections let a new column reuse, modify, or ignore features learned by earlier columns.The new column can combine prior representations with newly learned computations.
- Forgetting prevention: Frozen earlier parameters and one-way lateral connections prevent interference with prior tasks during training.Earlier columns are not affected by newly learned features in the forward pass.
- Adapters: Adapters use nonlinear lateral connections to improve initial conditioning and reduce dimensionality.For dense layers, the adapter is a single-hidden-layer MLP; convolutional layers use 1 × 1 convolutions for dimensionality reduction.
- Limitations: The approach grows in parameter count as the number of tasks increases, and selecting an inference column requires the task label.The paper notes that only a fraction of added capacity is used and suggests pruning or online compression as future directions.
3 Transfer Analysis
The paper measures transfer by perturbing representations or estimating their Fisher sensitivity. These measures quantify how strongly features and layers contribute to the target policy's output.
- Average Perturbation Sensitivity: Average Perturbation Sensitivity injects Gaussian noise at isolated architectural points and measures the resulting performance impact.A significant performance drop indicates reliance on the perturbed feature map or layer.
- Average Fisher Sensitivity: Average Fisher Sensitivity approximates perturbation sensitivity using a modified diagonal Fisher Information matrix over normalized activations.The matrix is interpreted as policy sensitivity to small representation changes.
- Sensitivity aggregation: The Fisher expectation uses the joint state-action distribution induced by the progressive network on the target task.Layer-level AFS sums feature sensitivities within a layer and column.
- Interpretation: AFS and APS estimate how much the target network relies on each feature or column at a given layer.Together they support analysis of where transfer occurs within the learned policy.
4 Related Literature
The related literature spans transfer and multitask reinforcement learning, pretraining and finetuning, and incremental architectures that support feature reuse or reduce forgetting. Progressive networks differ by adding capacity as new tasks are introduced.
- Transfer and multitask reinforcement learning: Prior transfer and multitask reinforcement-learning methods include approaches based on linear models and newer deep-RL methods.The paper situates progressive networks within longstanding transfer and multitask challenges in AI research.
- Finetuning: Pretraining and finetuning has been applied to transfer learning and to Atari reinforcement learning, with faster learning on some games but not others.The actor-mimic approach fine-tuned a multitask DQN network on new Atari games.
- Distinction: Progressive networks differ from finetuning by adding capacity as new tasks are introduced.This distinction is stated in the comparison with the finetuning direction.
- Incremental architectures: Incremental and constructive architectures have addressed forgetting, concept drift, and feature transfer through mechanisms such as added feature extractors or feature augmentation.The paper relates progressive networks to cascade-correlation, auto-encoders, and other deep architectures.
5 Experiments
The experiments evaluate progressive networks across synthetic Pong variants, Atari games, and 3D maze tasks, comparing them with transfer-learning baselines. Across these domains, progressive networks generally improve transfer while sensitivity analysis shows that successful transfer combines reused and newly learned features.
- Pong Soup: Progressive networks outperform full-model finetuning on Pong variants in both median and mean transfer score, with the larger mean difference indicating stronger gains when source and target domains are compatible.The comparison uses baseline 3, which finetunes the whole pretrained model, and progressive networks with prior columns retained.
- Detailed analysis: Pong transfer reuses different parts of the feature hierarchy: H-Flip reuses low- and mid-level vision but relearns policy-relevant fully connected features, whereas Zoom reuses low-level vision and learns new mid-level features.The analysis uses Average Fisher Sensitivity to identify relative dependence on columns within layers.
- Atari Games: Atari progressive networks show positive transfer on 8 of 12 target tasks, compared with 5 of 12 for full-model finetuning, while additional columns can improve transfer for some sequences.For Seaquest-to-Gopher, progressive networks avoid the negative transfer observed with baseline 3; additional columns improve Seaquest–River Raid–Pong to Boxing transfer.
- Detailed Analysis: In Atari, the most positive transfer occurs when previously learned features are augmented by new features, whereas the most negative transfer coincides with reliance on old convolutional features without new visual learning.Across 72 three-column networks, positive transfer appears at an intermediate level of new visual-feature learning.
- Labyrinth: Progressive networks yield more positive transfer than the baselines on Labyrinth tasks, especially on difficult games where the baselines struggle.Baseline 2 can show negative transfer on easy tasks because it cannot learn new low-level visual features when reward items change.
6 Conclusion
Progressive networks are presented as a step toward continual learning, combining transfer across tasks with robustness to incompatible features and no destructive forgetting.
- Progressive networks demonstrate potential for continual learning across three reinforcement-learning domains, including Atari tasks that can be orthogonal or adversarial.
- The architecture exploits transfer for compatible source and target domains while remaining robust to harmful features learned from incompatible tasks.
- Positive transfer increases with the number of columns, supporting the constructive rather than destructive nature of the progressive architecture.
A Perturbation Analysis
The perturbation analysis estimates which source-column representations materially contribute to target-task performance and corroborates Fisher-based sensitivity measurements.
- The appendix presents perturbation analysis as an intuitive but slower method for analysing transfer across tasks.
- Gaussian noise is injected into post-ReLU hidden representations, and its average effect on game score is measured over 10 episodes.The analysis adds noise across all features of a given layer at a coarse scale.
- Average Perturbation Sensitivity measures the precision of injected noise that produces a 50% performance drop.
- APS is normalized across columns within each layer and interpreted as each column’s responsibility for final performance.
- APS scores for two-column Pong-variant networks clearly corroborate the Average Fisher Sensitivity results.
B Compressibility of Progressive Networks
Progressive networks become increasingly sparse in their use of lateral features as tasks accumulate, suggesting that their parameter growth may be reducible through pruning.
- The number of hidden units and feature maps grows linearly with columns, while parameters grow quadratically.
- As columns increase, the average AFS spectrum becomes sparser, indicating reliance on a smaller proportion of source-column features.The analysis concatenates and sorts per-feature-map AFS values before averaging across networks.
- Features in the final added column become less important overall and exhibit a sparser AFS spectrum as the network grows.The declining area under the graph indicates reduced overall importance of the new column’s features.
- These findings suggest that significant pruning of lateral connections may contain the quadratic growth of parameters.
D Learning curves
The learning-curve evaluation compares progressive networks with Seaquest-pretrained baselines across Atari transfer sequences and target games. Overall, the finetuning baseline performs well, while additional prior-task features sometimes improve transfer, including in Boxing.
- Figure 11 compares two-, three-, and four-column progressive networks with Seaquest-pretrained finetuning and single-column baselines.
- Overall, the Seaquest-pretrained finetuning baseline performs well across the Atari target-game learning curves.
- Additional features learned from more previous tasks sometimes help transfer, with Boxing given as an example.
- Figures 12 and 13 compare progressive networks and baselines after learning standard Pong or Maze Y first.
E Labyrinth
The Labyrinth evaluation uses 3D maze foraging tasks that vary maze layouts, item types, and reward sparsity, with training curves reported after learning standard Pong or Maze Y.
- The 3D maze foraging tasks reward collecting apples and strawberries while penalizing mushrooms and lemons.Episodes end when all positive rewards are collected or after a fixed time interval.
- Maze levels vary in layout, item composition, and reward sparsity.
- The task set includes corridors, U-shaped and Ω-shaped tracks, a square room, and M- and Y-shaped mazes.The listed tracks differ in turn geometry, item placement, and whether the agent must seek or avoid items.
- Figures 12 and 13 report transfer training curves after learning standard Pong first or Maze Y first, respectively.