Source-linked AI summary
Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL
Anusha Nagabandi, Chelsea Finn, Sergey Levine
TL;DR
Deep neural networks represent complex functions but lack rapid online adaptation to changing conditions. MOLe combines meta-trained online SGD with EM and a Chinese restaurant process to maintain task-specific models. In model-based RL experiments, it outperformed prior methods and adapted across non-stationary tasks while recalling previously seen tasks.
Problem
Deep neural networks lack the rapid online adaptation needed when task distributions change and task identities are unknown.
Method
MOLe meta-trains neural networks for online SGD and uses EM with a Chinese restaurant process to maintain a mixture of task models.
Results
MOLe outperformed prior methods and demonstrated online task creation, adaptation to out-of-distribution tasks, and recall of previously seen tasks in model-based RL.
Takeaways & Limitations
The approach supports continuous adaptation and task recall in non-stationary model-based RL settings, with a broader stated applicability to streaming online learning.
Takeaways & Limitations
The online inference procedure is approximate because parameter updates can alter task probabilities assigned at earlier time steps.
Abstract
from arXiv · showhide
Humans and animals can learn complex predictive models that allow them to accurately and reliably reason about real-world phenomena, and they can adapt such models extremely quickly in the face of unexpected changes. Deep neural network models allow us to represent very complex functions, but lack this capacity for rapid online adaptation. The goal in this paper is to develop a method for continual online learning from an incoming stream of data, using deep neural network models. We formulate an online learning procedure that uses stochastic gradient descent to update model parameters, and an expectation maximization algorithm with a Chinese restaurant process prior to develop and maintain a mixture of models to handle non-stationary task distributions. This allows for all models to be adapted as necessary, with new models instantiated for task changes and old models recalled when previously seen tasks are encountered again. Furthermore, we observe that meta-learning can be used to meta-train a model such that this direct online adaptation with SGD is effective, which is otherwise not the case for large function approximators. In this work, we apply our meta-learning for online learning (MOLe) approach to model-based reinforcement learning, where adapting the predictive model is critical for control; we demonstrate that MOLe outperforms alternative prior methods, and enables effective continuous adaptation in non-stationary task distributions such as varying terrains, motor failures, and unexpected disturbances.
1 INTRODUCTION
The paper targets rapid continual adaptation of deep models to changing, unannounced tasks. MOLe combines online SGD, a task mixture, and meta-learning for model-based reinforcement learning.
- Motivation: Human and animal learning combines complex skill acquisition with rapid adaptation and recall under changing conditions.The motivation includes adapting to different surfaces, unexpected perturbations, and previously encountered disturbances.
- Problem setting: The online learner must infer task identities, decide when to create tasks, and decide when to update existing models.Task boundaries are not provided explicitly.
- Related limitation: Prior MAML-based model-based RL adapts from a prior using only the k most recent time steps, without carrying adaptation forward.The paper characterizes this batch-mode setting as restrictive for continuous online learning and tasks outside the training distribution.
- Approach: MOLe uses online SGD and a Chinese restaurant process to maintain and expand a mixture of task-specific neural-network models.The mixture can instantiate new tasks as the stream changes.
- Evaluation: The method is evaluated on simulated robotic tasks involving disturbances, environmental changes, and motor failures.The experiments include a half-cheetah and a hexapedal crawler.
2 RELATED WORK
Related work spans online and continual learning, meta-learning, and model-based RL, but prior approaches generally do not address continual online adaptation with unknown task separation.
- Online learning: Online learning includes gradient-update and probabilistic-filtering methods, but deep neural networks are difficult to train effectively with online SGD.High-dimensional models typically rely on batch-mode training.
- Continual learning: Continual learning addresses non-stationary task distributions and often emphasizes avoiding catastrophic forgetting.Online learning additionally emphasizes rapid adaptation during streaming data.
- Continual learning: Prior continual-learning work often studies small task sets, limiting opportunities to recover structure for fast few-shot adaptation.The cited work generally does not assume enough tasks for realistic fast online learning.
- Meta-learning: Most recent meta-learning methods learn one task at a time, whereas this work addresses non-stationary tasks without known boundaries.The paper distinguishes its runtime accumulation of task-specific mixture components from prior approaches.
3 PROBLEM STATEMENT
The paper formalizes online prediction when task identities are unknown and change over time. The learner must continually adapt to streaming data while balancing generalization and specialization.
- Formal setting: At each time step, the model predicts from x_t, receives y_t, and adapts to improve prediction on x_t+1.The labels arise from a task-conditioned distribution P(Y_t|X_t, T_t).
- Formal setting: Tasks change over time, but the learner does not observe the identity of the current task.Unknown tasks may represent robot malfunctions, user preferences, or unexpected events.
- Objective: The setting requires continual adaptation to streaming data while trading off generalization and specialization.This formulation covers multiple online learning problems.
- Application: In the model-based RL application, x_t is a state-action pair and y_t is the next state.The derivation remains general to arbitrary online prediction problems.
4 ONLINE LEARNING WITH A MIXTURE OF META-TRAINED NETWORKS
MOLe performs online inference over a growing mixture of neural-network models for unknown tasks. EM assigns task responsibilities, SGD updates models, and a CRP prior enables new task creation without storing past data.
- Task mixture: The posterior task distribution combines each model’s predictive likelihood with its prior task probability.This posterior supplies the responsibilities used for prediction and updating.
- Method overview: Each task has neural-network parameters initialized from a shared prior, while the number of tasks is unknown.The system begins with one task and adds tasks as needed.
- Online inference: EM estimates task probabilities in the E step and updates every model in the M step according to inferred responsibilities.Parameters receive one gradient step per time step.
- Parameter updates: One gradient update per time step provides the online parameter update for each task model.The update is weighted by inferred task responsibilities and uses the newest data.
- Task mixture: The CRP prior assigns probabilities to existing tasks and enables new tasks to be instantiated during a trial.The expected task counts determine existing-task probabilities, while α controls new-task instantiation.
- Approximate inference: The streaming procedure avoids storing previously seen data points, but its single-step EM procedure is an approximation.The authors note that multiple EM steps remain possible while staying fully online.
5 META-LEARNING THE PRIOR
MOLe meta-trains a prior that supports gradient-based adaptation, then extends adaptation over time by maintaining and updating task-specific neural-network models. This addresses the limitations of prior k-shot methods, which repeatedly adapt from the prior without continued specialization.
- Continual adaptation: MOLe performs continual online adaptation by updating a mixture of meta-trained networks with online SGD and expectation maximization.The online procedure maintains task-specific parameters and updates their task probabilities as data arrive.
- MAML prior: MAML meta-trains a deep-network initialization that can adapt to new tasks using a few datapoints and gradient steps.The resulting parameters serve as a prior for test-time fine-tuning.
- Task evolution: The method uses a Chinese restaurant process prior to instantiate new task models as needed while retaining existing models for later adaptation.Task assignments are updated online rather than supplied as explicit task boundaries.
- Continual adaptation: Unlike prior k-shot model-based meta-RL methods, MOLe carries adaptation forward in time and permits further specialization beyond the meta-learned prior.Prior methods always adapted directly from the prior and did not allow continued adaptation or specialization.
- Meta-training choice: A modified MAML objective incorporating EM-weighted updates did not substantially improve results, suggesting standard MAML can enable effective continuous online adaptation.The authors note that longer-term meta-training could produce a larger difference.
6 APPLICATION TO MODEL-BASED RL
MOLe is applied to model-based reinforcement learning by estimating task-dependent dynamics from recent state-action histories. The predictive model supports planning, while the agent’s model-dependent actions determine the subsequent data stream.
- Dynamics modeling: MOLe estimates the task-conditioned dynamics distribution p(s_t+1|s_t, a_t, T_t) with a predictive model p_θ.The latent task variable represents system settings, external details, and environmental perturbations.
- Online data: Each online input concatenates K previous states and actions, with corresponding next states as targets for the predictive model.Using recent transitions provides a larger update batch and dampens noisy high-frequency updates.
- Dynamics modeling: The predictive model represents the transitions as independent Gaussian distributions, with neural-network means and constant variance.The mean dynamics function is implemented as a three-hidden-layer ReLU network with 500 units per layer.
- Task updates: A separate set of K nearby datapoints is used to calculate a new task parameter, preventing evaluation and parameter calculation from using the same data.The new parameter may or may not be added to the task distribution.
- Model-based control: After selecting the most likely task, the controller uses the corresponding predictive model to plan future actions that maximize reward.Because the controller’s action choice depends on estimated model parameters, the model also influences which next data point is observed.
- Meta-training: MAML is performed inside model-based RL, using controller-generated on-policy rollouts to supply data for meta-training.The model and controller interact repeatedly during meta-training.
7 EXPERIMENTS
Experiments in MuJoCo test MOLe on half-cheetah and crawler agents under fixed and changing terrains, motor malfunctions, and leg crippling. Across dynamically changing tasks, MOLe supports online adaptation, task recognition, and recall, while baselines expose the need for adaptation, meta-learned initialization, and protection against forgetting.
- Experimental setup: Experiments evaluate MOLe on half-cheetah and hexapedal crawler agents across sudden, gradual, and extrapolated task changes.The experiments use MuJoCo agents and compare methods with and without meta-training.
- Terrain slopes on half-cheetah: Model-based RL and model-based RL with online gradient updates perform poorly on out-of-distribution terrains, indicating the need for adaptation and a meta-learned initialization.The baselines were trained on the same data received by the meta-trained model.
- Terrain slopes on half-cheetah: MOLe uses a single latent task variable for changing terrain slopes, suggesting that its meta-learned parameter space supports skill transfer without separate task parameters.The discovered task representation does not correspond to human-distinguishable categorical labels.
- Half-cheetah motor malfunctions: Under changing motor malfunctions, continuously adapting methods overfit recent data and forget prior skills, whereas MOLe recognizes new tasks and recalls old ones online.MOLe performs recognition and adaptation without a bank of past data or human-specified task categories.
- Crippling of end effectors on six-legged crawler: Continued gradient updates on the crawler were comparable to MOLe initially but became 200% lower during the final 500 steps of normal walking.The result illustrates the detrimental effect of updating without separate task specialization or adaptation.
- Crippling of end effectors on six-legged crawler: For the crawler, all methods perform well on fixed tasks, but only MOLe learns effectively when leg-crippling tasks change dynamically within a trial.MOLe recognizes alternating normal and crippled periods and improves when the same tasks recur.
8 DISCUSSION
The paper presents an online learning method for non-stationary, multi-task settings that combines meta-trained neural networks with probabilistic task updates. In model-based RL, it adapts to new tasks, develops task representations, and recalls previously seen tasks.
- MOLe handles non-stationary, multi-task trials by adapting neural network models directly with SGD while EM and a Chinese restaurant process maintain task distributions.
- Meta-training with MAML and probabilistic test-time updates enables effective online learning with neural networks.
- In model-based RL experiments, the method adapted simulated robots to varied and unexpected tasks.
- MOLe developed its own notion of task, adapted away from the prior when needed, and recalled previously encountered tasks.
- Although evaluated in model-based RL, the method is described as applicable to other streaming and online learning settings.
A TEST-TIME PERFORMANCE VS TRAINING DATA
Test-task performance improves as meta-trained models receive more training data. The reported performance is normalized per agent on tasks unseen during training.
- Performance on unseen test tasks improves as meta-trained models are trained with more data.
- Test-task performance numbers are normalized per agent between 0 and 1.
B HYPERPARAMETERS
The experiments use fixed neural-network dynamics-model and controller settings, alongside listed train-time and run-time hyperparameters. The controller samples 1000 action sequences of horizon H=10 at each time step.
- The dynamics model has three hidden layers of dimension 500 with ReLU nonlinearities.
- Random-shooting MPC samples 1000 candidate action sequences, each with horizon H=10, at every time step.
- The listed parameters distinguish tasks sampled per training iteration from total time steps collected per iteration.
- Separate tables report hyperparameters for train-time and run-time.
C CONTROLLER
The controller uses the learned dynamics model and reward function to evaluate randomly sampled action sequences. It executes only the first action of the best sequence, then replans using updated state information.
- At each time step, the controller randomly generates N candidate action sequences with H actions each.
- The learned dynamics model predicts the states resulting from each candidate action sequence.
- The reward function selects the candidate sequence with the highest predicted reward.
- MPC executes only the first action of the selected sequence and replans at the next time step.
- Replanning with updated state information can compensate for model inaccuracies by preventing errors from accumulating.