Source-linked AI summary
Expert Gate: Lifelong Learning with a Network of Experts
Rahaf Aljundi, Punarjay Chakravarty, Tinne Tuytelaars
TL;DR
Lifelong learning must add tasks sequentially without retaining prior data while determining which specialist model to deploy at test time. Expert Gate uses task autoencoders to select experts and guide transfer, achieving competitive task assignment and improved video-prediction results across the evaluated settings.
Problem
Lifelong learning lacks a data-free solution for selecting the most relevant expert at test time as tasks are learned sequentially.
Method
Expert Gate adds specialist experts sequentially and uses undercomplete task autoencoders to select the relevant expert and guide prior-model transfer without storing previous-task data.
Results
Expert Gate’s autoencoders assign test samples to relevant tasks equally accurately as a discriminative classifier, while video prediction outperforms sequential and joint training.
Takeaways & Limitations
Autoencoders provide both test-time expert selection and training-time task-relatedness information for lifelong learning across image classification and video prediction.
Takeaways & Limitations
The autoencoder-gating analysis relies on an unregularized one-layer under-complete autoencoder and its reconstruction-error criterion.
Abstract
from arXiv · showhide
In this paper we introduce a model of lifelong learning, based on a Network of Experts. New tasks / experts are learned and added to the model sequentially, building on what was learned before. To ensure scalability of this process,data from previous tasks cannot be stored and hence is not available when learning a new task. A critical issue in such context, not addressed in the literature so far, relates to the decision which expert to deploy at test time. We introduce a set of gating autoencoders that learn a representation for the task at hand, and, at test time, automatically forward the test sample to the relevant expert. This also brings memory efficiency as only one expert network has to be loaded into memory at any given time. Further, the autoencoders inherently capture the relatedness of one task to another, based on which the most relevant prior model to be used for training a new expert, with finetuning or learning without-forgetting, can be selected. We evaluate our method on image classification and video prediction problems.
1. Introduction
Expert Gate addresses lifelong learning without retaining previous-task data by adding specialist experts sequentially and automatically selecting the relevant expert at test time. Its autoencoders also guide knowledge transfer between related tasks, and the system is evaluated on image classification and video prediction.
- Sequentially adding tasks can cause catastrophic forgetting, degrading performance on earlier tasks after fine-tuning on new data.
- Jointly training tasks requires retaining all previous data and can introduce negative inductive bias, weaken specialist representations, and require repeated network retraining.
- Expert Gate builds a Network of Experts, adding a specialist model for each new task while transferring knowledge from prior models without storing all previous data.
- A gating mechanism uses the test sample to activate the appropriate expert, reducing the need to load all models into limited GPU memory.
- Undercomplete autoencoders learn task-specific representations and select an expert by comparing reconstruction errors for the test sample.
- The autoencoders also estimate task relatedness to select a prior model and transfer strategy, including fine-tuning or learning without forgetting.
- The system is evaluated on image classification and video prediction problems.
2. Related Work
The related work spans multi-task learning, multiple-model architectures, and lifelong learning methods that transfer knowledge while avoiding catastrophic forgetting. Expert Gate differs by using task autoencoders to automate task relatedness and expert selection without retaining prior training data.
- Multi-task learning: Multi-task learning jointly trains multiple tasks to exploit inductive bias, often through a shared model.
- Multi-task learning: Prior work identifies related tasks through mutual-information clustering or by grouping task models whose parameters are close in an original or lower-dimensional space.
- Multi-task learning: Expert Gate instead uses fast-to-train task autoencoders to identify related tasks.
- Multiple models for multiple tasks: Multiple-model architectures use separate expert networks and a gating network, but earlier approaches required each sample to pass through every expert.
- Lifelong learning without catastrophic forgetting: Lifelong-learning methods transfer knowledge between sequential tasks while seeking to prevent catastrophic forgetting and avoid storing all lifetime training data.
- Lifelong learning without catastrophic forgetting: Progressive and modular networks add a network for each task with lateral connections, but task or column selection at test time remained manual.
- Lifelong learning without catastrophic forgetting: Expert Gate uses an autoencoder to determine which model, and consequently which column, should be selected for each test sample.
3. Our Method
Expert Gate learns task-specific autoencoders alongside sequentially added expert models, using reconstruction and relatedness signals to route test samples and guide transfer from prior tasks.
- Autoencoder gating: Each task receives a specialized expert and an autoencoder that captures its data characteristics without retaining previous task data.The learned autoencoder represents each task in a lower-dimensional subspace used for later task recognition.
- Autoencoder gating: At test time, the autoencoder with the lowest reconstruction error selects the expert associated with the most relevant task.A softmax converts reconstruction errors into task confidence values, and the most confident expert is loaded; overlapping tasks may activate multiple experts using a threshold.
- Autoencoder gating: The gate addresses GPU memory limits by loading only the selected expert instead of all task-specific models.This gating mechanism uses the test sample to decide which expert to activate as the number of task specializations grows.
- Task relatedness: Task relatedness is estimated by comparing current-task reconstruction errors from the current and previous task autoencoders, producing an asymmetric relation.The current task’s validation data is used because previous task data are unavailable.
- Task relatedness: The most related prior task supplies the model for learning a new expert, while a relatedness threshold chooses between LwF and fine-tuning.LwF is used above the threshold; otherwise the prior model is fine-tuned, because enforcing old outputs can hurt when tasks are insufficiently related.
4. Experiments
Experiments evaluate Expert Gate on sequential image-classification and video-prediction tasks, comparing it with joint, fine-tuned, LwF, confidence-based, and discriminative baselines. Across these settings, the gate preserves task-specific expertise while selecting relevant experts without requiring simultaneous access to prior-task data.
- Experimental setup: Expert Gate is evaluated on sequential image classification, expanded from three to six tasks, and on video prediction.The experiments include baseline comparisons, gate-behavior and task-relatedness analyses, and a video-prediction evaluation.
- Image classification baselines: Sequential fine-tuning causes catastrophic forgetting, while single-model LwF remains inferior to maintaining exclusive expert models across older and newer tasks.The paper attributes LwF degradation to accumulated errors, degraded soft targets, and differences in task relatedness.
- Image classification baselines: Expert Gate always identifies ImageNet as the most related prior task in the reported datasets and selects LwF or fine-tuning accordingly for new experts.For the six-task experiment, LwF is selected for Actions, while Aircrafts and Cars are fine-tuned.
- Gate analysis: Expert Gate achieves average performance superior to jointly trained models in the six-task experiment, where joint training can suffer negative inductive bias between tasks such as Scenes and Cars.Joint training assumes all previous data remains available, whereas Expert Gate uses sequentially learned experts.
- Gate analysis: Expert Gate assigns test samples to relevant tasks as accurately as a discriminative classifier despite never having simultaneous access to previous-task data.The discriminative classifier is evaluated as an upper bound because it stores previous-task data; with 2000 stored samples per task, it approaches the gate’s accuracy.
- Gate analysis: 68.2% average accuracy results when a 0.1 threshold allows multiple experts, with 3.7% of test samples analyzed by multiple models and a further increase of 0.9%.These confusion cases arise when even humans have difficulty deciding which expert should activate, and evaluation is limited to the corresponding task label.
- Video prediction: On video prediction, Expert Gate outperforms both sequential fine-tuning and joint training, while sequential fine-tuning exhibits catastrophic forgetting.The experiment uses autoencoders only for gating, and video-prediction error is measured by average pixel-wise L1 distance, where lower is better.
5. Conclusions and Future Work
Expert Gate addresses lifelong learning by selecting relevant experts without storing prior-task data, while using task relatedness to guide knowledge transfer. It outperforms state-of-the-art methods and joint training across the reported evaluations.
- Expert Gate selects the relevant expert at test time without requiring stored data from previous tasks.
- Its autoencoders distinguish tasks as accurately as a discriminative classifier trained on all data.
- The autoencoders identify the most related prior task and appropriate transfer method for training new experts.
- Expert Gate outperforms both state-of-the-art methods and joint training of all tasks simultaneously.