Source-linked AI summary
Encoder Based Lifelong Learning
Amal Rannen Triki, Rahaf Aljundi, Mathew B. Blaschko, Tinne Tuytelaars
TL;DR
Sequential lifelong learning must adapt to new tasks without catastrophic forgetting of earlier ones. The paper uses task-specific undercomplete autoencoders to preserve crucial representations while leaving other features adaptable, and reports reduced forgetting with improved performance over the state of the art across tested image-classification scenarios.
Problem
Lifelong learning seeks to adapt a single model to sequential tasks while retaining knowledge of previously seen tasks, without storing all earlier data or large parameter sets.
Method
The method trains an undercomplete autoencoder after each task and constrains reconstructions of its important features while allowing other representation dimensions to adapt to the new task.
Results
The method continually reduces forgetting compared to LwF while maintaining comparable new-task performance, achieving 54.9% on ImageNet versus 53.3% for LwF.
Takeaways & Limitations
Across tested image-classification sequences, the method improves performance over the state of the art, especially by better preserving earlier tasks.
Takeaways & Limitations
The current approach relies on new data, motivating autoencoders as data generators when that data does not represent previous distributions well.
Abstract
from arXiv · showhide
This paper introduces a new lifelong learning solution where a single model is trained for a sequence of tasks. The main challenge that vision systems face in this context is catastrophic forgetting: as they tend to adapt to the most recently seen task, they lose performance on the tasks that were learned previously. Our method aims at preserving the knowledge of the previous tasks while learning a new one by using autoencoders. For each task, an under-complete autoencoder is learned, capturing the features that are crucial for its achievement. When a new task is presented to the system, we prevent the reconstructions of the features with these autoencoders from changing, which has the effect of preserving the information on which the previous tasks are mainly relying. At the same time, the features are given space to adjust to the most recent environment as only their projection into a low dimension submanifold is controlled. The proposed system is evaluated on image classification tasks and shows a reduction of forgetting over the state-of-the-art
1. Introduction
Lifelong learning requires adapting a model to sequentially presented tasks without losing previously learned knowledge. Existing approaches trade off adaptation, retention, storage, or access to earlier data, motivating an autoencoder-based compromise.
- Motivation: Catastrophic forgetting occurs when sequentially learned models adapt to new tasks or environments but lose performance on earlier tasks.The introduction illustrates this risk using an agent that first localizes factory-product defects and later learns anomaly detection for new products.
- Motivation: Lifelong learning must adapt to new data from similar or different environments while retaining knowledge from previously seen tasks.
- Existing approaches: Feature extraction preserves the old representation but is conservative for the old task and suboptimal for the new one, whereas fine-tuning adapts but forgets prior knowledge.
- Existing approaches: Joint training can be optimal across tasks but requires all task data simultaneously, which can be difficult when datasets are large.
- Proposed approach: The proposed compromise trains an undercomplete autoencoder per task to capture important features and constrains their reconstructions while allowing remaining features to adapt.The autoencoder is trained after the task model, and Figure 1 presents the resulting model components and training scheme.
2. Related work
Related work addresses lifelong learning by preserving prior predictions or constraining important parameters, but these strategies face distributional or memory limitations.
- Multitask and lifelong learning: Multitask learning integrates knowledge by jointly training on multiple tasks, but requires data from all tasks during training.
- Learning without Forgetting: Learning without Forgetting preserves previous-task outputs through knowledge distillation while using a shared convolutional network and task-specific classification layers.
- Elastic weight consolidation: Elastic weight consolidation constrains parameters important to an earlier task using a Fisher-information-weighted penalty during training on a second task.
- Elastic weight consolidation: Elastic weight consolidation may restrict learning to a neighborhood of one earlier minimizer and requires storing a large number of parameters.
3. Overcoming forgetting with autoencoders
The method preserves task-relevant feature information through an undercomplete autoencoder while allowing other features to adapt to new tasks. It extends to sequential task learning and controls representation changes without storing previous task data.
- Model setup: The model decomposes each task function into feature extraction, shared task operations, and task-specific operators.F extracts features; T is shared across tasks, while Tt is task specific.
- Limitations of LwF: LwF can lose information when the distributions of successive tasks differ substantially, because it evaluates old-task behavior on new-task data.The paper reports no direct guarantee under unrelated distributions and cites empirical performance decreases.
- Informative feature preservation: The proposed approach learns a submanifold containing the features most informative for the first task and constrains new representations only through their projection onto it.This preserves information useful for the first task while leaving the remaining feature components free to adjust.
- Learning the informative submanifold: An undercomplete autoencoder captures the lower-dimensional feature submanifold by reconstructing task representations, while its objective also preserves information needed by the task operator.The loss balances reconstruction and task performance through the hyper-parameter λ.
- Shared task operator: For shared task operators, the method adds representation-preservation constraints to the new-task loss alongside the existing LwF loss.This shared architecture reduces memory growth when adding tasks.
- Training procedure: The procedure trains an autoencoder after each task, with memory growing linearly across tasks but requiring only around 1.5% of an AlexNet base model per autoencoder.The method therefore stores a small fraction of the global model for each task.
4. Experiments
Experiments evaluate the method on sequential image-classification tasks with two- and three-task settings, varying the initial dataset and comparing against LwF and baselines. The results show reduced forgetting while retaining comparable performance on newly learned tasks, with separate fully connected layers offering stronger performance at higher memory cost.
- Experimental setup: Experiments use AlexNet on sequential image-classification tasks with two- and three-task sequences, beginning from either ImageNet or Flowers.The evaluation includes large- and small-initial-dataset settings and reports results on Scenes, Birds, Flowers, and ImageNet validation data.
- Experimental setup: The method is compared with Learning without Forgetting, Finetuning, Feature extraction, and a variant with separate fully connected layers.The shared model normally uses convolutional layers and earlier fully connected layers jointly, while the separate-FCs variant allocates task-specific fully connected layers.
- Two-task results: Finetuning performs best on the new task but produces the most forgetting on the previous task, whereas Feature extraction preserves the previous task but performs suboptimally on the new one.These baselines illustrate the trade-off between adapting to the new task and retaining earlier knowledge.
- Two-task results: Ours with separate FCs achieves the best compromise between the two tasks, while the complete shared model systematically outperforms LwF on the previous task with similar average performance on the second task.The separate-FCs variant has higher capacity but its memory consumption increases more rapidly as tasks are added.
- Two-task results: Starting from Flowers produces larger accuracy differences: separate FCs performs considerably better on the second task than Ours and LwF while preserving the previous task, whereas Finetuning suffers severe forgetting.Across the Flowers-starting experiments, Ours reduces forgetting relative to LwF while achieving similar average performance on the second task.
- Sequential results: In the three-task ImageNet → Scenes → Birds sequence, Ours continually reduces forgetting compared with LwF while maintaining comparable performance on the new task, achieving 54.9% on ImageNet versus 53.3% for LwF.Finetuning drops ImageNet performance from 57% to 37.5% after training on the third task.
5. Conclusions and future work
The method addresses lifelong learning from only the most recent environment by preserving task-critical features rather than previous optimal weights. Across tested image-classification sequences, it improves performance over the state of the art, especially for old-task preservation, while future work targets weak coverage of previous distributions.
- The method learns task-critical feature submanifolds with undercomplete autoencoders to reduce forgetting across sequential tasks.It controls distances between task representations instead of preserving previous task weights.
- Image-classification experiments on two- or three-task sequences improve performance over the state of the art in all tested scenarios.Experiments begin from either small or large datasets.
- The experiments show better preservation of old tasks under the proposed representation-control strategy.
- A future direction is using autoencoders as data generators instead of relying on new data.This could strengthen the method when new data does not represent previous distributions well.
A. Analysis of the main method
The theoretical analysis assumes Lipschitz-continuous training functions and losses, a condition satisfied by several commonly used choices. It explicitly notes that squared and exponential losses fall outside this assumption.
- The derivations assume that all functions involved in model training are Lipschitz continuous.
- Sigmoid, ReLU, linear, softmax, logistic, and hinge functions or losses satisfy the stated continuity condition.
- Squared and exponential losses are not Lipschitz continuous, although the paper describes this as a limited practical concern.
- The analysis introduces Lipschitz continuity through a constant K bounding function-value changes by input-distance changes.
A1. Relation between Encoder based Lifelong Learning and joint training
The analysis relates encoder-based lifelong learning to joint training by decomposing their risk difference into controllable and uncontrollable terms. Autoencoder and distillation constraints preserve prior-task behavior while one feature component remains relaxed for adaptation.
- Joint training with data from both tasks is the ideal reference, whereas the proposed objective uses only second-task data.
- The difference from a joint-training risk can be controlled through knowledge distillation and five terms, A7 through A11.
- Terms on the inequality’s right-hand side that remain unchanged during task-2 training cannot be directly controlled in that phase.The data shift in A6 motivates adding knowledge distillation as in Learning without Forgetting.
- Autoencoder training fixes A8, while A9 measures distance within the represented manifold and is empirically smaller than raw input distance.
- The second proposed constraint controls A10 during training, while A11 is relaxed to let features adapt to task 2.
- Controlling A11 more tightly could preserve task-1 performance more strongly but risks preventing convergence on task 2.
- During training, task-2 optimization initially dominates, then distillation and encoder terms push the representation back toward the task-1 representation.
A2. Empirical study: F and r are contractive
The empirical study tests whether feature and reconstruction mappings contract distances between inputs from different distributions. Repeated Gaussian-distribution trials on Flowers and ImageNet are summarized in tables whose entries decrease from samples to features to reconstructions.
- The experiment compares input, feature, and reconstruction distances using the ℓ2 norm to test whether F and r are contractive.
- The procedure generates inputs from two random Gaussian distributions, computes features and autoencoder reconstructions, and records mean squared errors.
- Tables A1 and A2 report Flowers- and ImageNet-trained mappings, respectively, and show entries decreasing from left to right.The captions state that this decrease verifies contractive mappings.
- Mean MSE is computed over repeated trials using different distribution pairs, with separate settings for relatively small and larger sample distances.
A3. Multiple task scenario
As more tasks are added, the gap from joint-training grows because each step introduces an additional irreducible discrepancy. The method leaves selected distances relaxed to preserve flexibility for learning the new task.
- Each added task introduces another divergence from the joint-training objective, so the difference grows with the number of tasks.
- An extra irreducible discrepancy between feature reconstructions on consecutive task data is added at every step.
- The previous-task autoencoders are trained using their corresponding feature extractors.
- Some remaining loss differences are neither directly controlled nor guaranteed to decrease during training.
- Relaxing selected distances is an important degree of freedom because controlling them could prevent convergence on the new task.
B. Additional experiments
Additional experiments evaluate the method with VGG-verydeep-16 on sequential ImageNet and MIT Scenes classification, comparing it with LwF. The method preserves ImageNet performance better, uses relatively little additional memory, and provides stronger regularization on Scenes, although overfitting remains a concern for the large model.
- Tested scenario: The experiment uses VGG-verydeep-16 for sequential classification from ImageNet to MIT Scenes and compares the method with LwF.Results use ImageNet validation data and Scenes test data, with more than 1 million ImageNet training images and 5,360 Scenes samples.
- Architecture: The autoencoder architecture has two layers with an intermediate sigmoid and a 300-entry code.
- Autoencoder size: The autoencoder remains very small compared with the global model, although its size increases with feature length.Table B1 compares the memory required by AlexNet and VGG-verydeep-16 with the corresponding autoencoder memory.
- Results: After 20 epochs, ImageNet performance is 1% higher with the proposed method than with LwF.
- Results: The code loss appears to regularize Scenes training, producing slightly higher training error and better generalization.
- Conclusion: The observed autoencoder convergence and improvement over LwF do not depend on the tested architecture.