Source-linked AI summary
Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments
Maruan Al-Shedivat, Trapit Bansal, Yuri Burda, Ilya Sutskever, Igor Mordatch, Pieter Abbeel
TL;DR
Nonstationary environments require agents to adapt from limited interaction, but reactive fine-tuning is poorly suited to this few-shot setting. The paper develops gradient-based meta-learning for continuous adaptation, evaluates it in locomotion and RoboSumo, and finds more efficient adaptation than baselines, with meta-learners emerging as the fittest agents.
Problem
Nonstationarity limits interaction before environmental properties change, creating a few-shot adaptation problem that makes standard fine-tuning impractical.
Method
The paper models nonstationarity as a sequence of stationary tasks and meta-learns gradient-based policy updates using information from consecutive tasks.
Results
Meta-learned adaptation strategies clearly dominate other methods in the few-shot regime across single-agent locomotion and multi-agent RoboSumo settings.
Takeaways & Limitations
In population experiments, agents with meta-learned adaptation strategies end up being the fittest according to the reported competition and evolution results.
Takeaways & Limitations
The approach relies on one-step-ahead updates, second-order derivatives that slow training by an order of magnitude, and policy gradients that are unlikely to work with sparse rewards.
Abstract
from arXiv · showhide
Ability to continuously learn and adapt from limited experience in nonstationary environments is an important milestone on the path towards general intelligence. In this paper, we cast the problem of continuous adaptation into the learning-to-learn framework. We develop a simple gradient-based meta-learning algorithm suitable for adaptation in dynamically changing and adversarial scenarios. Additionally, we design a new multi-agent competitive environment, RoboSumo, and define iterated adaptation games for testing various aspects of continuous adaptation strategies. We demonstrate that meta-learning enables significantly more efficient adaptation than reactive baselines in the few-shot regime. Our experiments with a population of agents that learn and compete suggest that meta-learners are the fittest.
1 INTRODUCTION
Nonstationary environments demand continuous adaptation from limited interaction, making standard fine-tuning impractical. The paper frames this challenge as meta-learning and evaluates gradient-based adaptation in locomotion and competitive multi-agent settings.
- Nonstationarity arises from changing dynamics, objectives, or concurrently learning agents, requiring adaptation during both training and execution.
- Limited interaction before environmental changes puts nonstationary learning in the few-shot regime, where sample-inefficient deep RL and fine-tuning methods become impractical.
- The paper treats a nonstationary environment as a sequence of stationary tasks and uses meta-learning to generalize adaptation from few examples.
- RoboSumo is a 3D simulated-physics environment for competitive pairs, with iterated adaptation games allowing both opponents to update between rounds.
- Across nonstationary locomotion and RoboSumo, meta-learned strategies dominate other adaptation methods in the few-shot regime; population experiments suggest they are fittest.
2 RELATED WORK
Continuous adaptation differs from broader lifelong and never-ending learning by targeting one changing task under limited execution-time interaction. The paper also positions RoboSumo as a harder multi-step, continuous-time setting where fast opponent adaptation is important.
- Lifelong learning transfers knowledge across sequential tasks while minimizing catastrophic forgetting, whereas never-ending learning repeatedly improves a growing task set.
- Continuous adaptation targets a single nonstationary task or environment rather than a general sequence of tasks.
- Unlike lifelong and never-ending learning, continuous adaptation occurs at execution time under limited data or interaction between environmental changes.
- RoboSumo differs from one-shot iterated games because each episode has multiple continuous-time steps and requires learning an intra-episodic controller.
- The proposed method follows meta-learning by learning a high-level procedure that generates a policy whenever the environment changes.
3 METHOD
The paper frames nonstationary environments as sequentially dependent tasks and meta-learns gradient-based adaptation updates that use limited experience from one task to prepare for the next. Training optimizes these updates over consecutive task pairs, while execution reuses past experience because repeated access to each task is usually unavailable.
- MAML: MAML constructs a task-specific policy from K trajectories collected under an initial policy, optimizing parameters for low subsequent task loss.The adaptation update is parameterized by θ, while the task-specific policy parameters are denoted φ.
- Training: Training samples pairs of consecutive tasks and jointly optimizes the initialization θ and adaptation step sizes α with gradient updates.The method can construct φ using multiple meta-gradient steps with adaptive step sizes.
- Probabilistic view: Nonstationary environments are modeled as sequences of stationary tasks whose transitions can be represented by a Markov chain.The task-level dynamics form an upper hierarchy over the MDP dynamics of individual tasks.
- Continuous adaptation: The sequential meta-loss uses trajectories from Ti to construct a policy that performs well on the upcoming task Ti+1.Starting from θ for each pair makes the objective equivalent to truncated backpropagation through time with a unit lag.
- Execution: At execution time, the agent keeps acting with πφ and reuses past experience to update φ for each new task.This differs from training because unbiased adaptation gradients would otherwise require interacting with each task twice.
4 ENVIRONMENTS
The paper evaluates continuous adaptation in changing locomotion and competitive multi-agent settings built with MuJoCo physics and multi-leg robot agents. RoboSumo extends evaluation to repeated games where both opponents can adapt between rounds.
- Environment suite: The environments cover episode-to-episode nonstationarity in locomotion and incremental policy changes during competitive RoboSumo games.All environments use MuJoCo physics and simple multi-leg robots.
- Locomotion: In locomotion, selected joint torques are scaled from 1 to 0 over 7 episodes to create nonstationarity.The six-leg agent observes body and leg state and receives reward proportional to speed in a fixed direction.
- RoboSumo: RoboSumo is a 1-vs-1 continuous-action environment with Ant, Bug, and Spider agents that differ in anatomy.Agents observe themselves, their opponent, joint states, velocities, and forces on their own bodies.
- Iterated adaptation games: Iterated adaptation games use repeated fixed-length episodes and allow both agents to update their policies between rounds.An agent should exploit repeated encounters with the same opponent while accounting for the opponent’s own adaptation.
- Reward shaping: RoboSumo rewards include +2000 for a winner, -2000 for a loser, and -1000 for each agent in a draw.Additional shaping rewards encourage early learning through positioning, movement, and contact-related behavior.
5 EXPERIMENTS
The experiments compare meta-learning and baseline adaptation strategies in nonstationary locomotion and competitive RoboSumo games, including few-shot adaptation, sample complexity, and population-level competition. Meta-learned strategies perform best in limited-data settings and consistently achieve higher skill, eventually dominating an evolved population.
- 5.1 THE SETUP: The study controls training and evaluation factors to compare naive, RL2, tracking, and meta-learning adaptation strategies across locomotion and RoboSumo.Locomotion uses held-out nonstationary environments, while RoboSumo uses pre-trained opponents and a common training curriculum.
- 5.1 THE SETUP: Experiments test behavior and sample complexity when agents receive only one or a few episodes before the environment changes.The amount of experience from the same environment is explicitly controlled before each change.
- 5.2 ADAPTATION IN THE FEW-SHOT REGIME AND SAMPLE COMPLEXITY: After 6-7 episodes, meta-updated policies perform much better than other methods in held-out nonstationary locomotion environments.The initial meta-learned policy is initially suboptimal, but becomes competitive after 1-2 episodes; tracking does not improve over no adaptation and can perform worse.
- 5.2 ADAPTATION IN THE FEW-SHOT REGIME AND SAMPLE COMPLEXITY: Within about 100 episodes against constantly improving opponents, meta-learned strategies usually improve win rates while baseline performance often deteriorates.Opponents receive 90 self-play episodes per iteration, whereas agents receive only 3 episodes per round.
- 5.2 ADAPTATION IN THE FEW-SHOT REGIME AND SAMPLE COMPLEXITY: Meta-learning maintains nearly constant performance as episodes per round increase, while tracking can learn against unseen opponents above 50 episodes per round.The authors attribute this pattern to a learned bias that helps with limited experience but limits use of additional data, given fixed three-step meta-updates.
6 CONCLUSION AND FUTURE DIRECTIONS
The paper presents gradient-based meta-learning for continuous adaptation in nonstationary environments and reports few-shot efficiency gains over baselines. It also identifies unresolved limitations involving update history, computational cost, and reward sparsity.
- Conclusion: Meta-learned adaptation rules were more efficient than baselines in few-shot nonstationary locomotion and competitive multi-agent settings.The approach was evaluated in both single-agent and competitive environments.
- Conclusion: Agents that meta-learned to adapt demonstrated the highest skill when competing against one another in iterated games.This result came from competitive interactions among agents that learned and competed.
- Method: The method treats nonstationarity as a sequence of stationary tasks and exploits dependencies between consecutive tasks during adaptation.The learned rule is intended to handle similar task transitions at execution time.
- Future directions: The problem remains far from solved, and the work is described as the first attempt to apply meta-learning in this setup.The authors explicitly characterize the approach as an initial effort.
- Limitations: The one-step-ahead meta-update could be extended to recurrent updates using the full interaction history, but current surrogate-loss updates require second-order derivatives and slow training by an order of magnitude.The authors also note that sparse rewards are problematic because the updates rely heavily on policy-gradient reward signals.
A DERIVATIONS AND THE POLICY GRADIENT THEOREM
The appendix formulates meta-learning as optimizing a task-specific policy produced from limited experience and derives its policy-gradient objective. The resulting gradient includes both the updated-policy term and a term accounting for trajectories collected under the original policy.
- Meta-learning objective: Meta-learning seeks parameters for a procedure that uses limited task experience to produce a policy suitable for solving that task.The procedure fθ both collects initial experience and constructs the task-specific policy.
- Meta-learning objective: The task objective marginalizes trajectories and adapted policy parameters so that the optimization depends on the initial parameters θ.Trajectories and policy parameters are treated as random variables in the construction.
- Policy-gradient derivation: The gradient contains a standard policy-gradient term for the updated policy and a second term for the original policy used to collect the adaptation trajectories.The second term arises because the collected trajectories depend on θ.
- Policy-gradient theorem: The gradient estimator is unbiased when the loss is the sum of discounted rewards, extending REINFORCE to the meta-learning setting.Value- or advantage-based losses can similarly extend the policy-gradient theorem.
- Policy-gradient theorem: The continuous-setting version changes the distributions used for the outer and inner expectations in the policy-gradient expression.The outer expectation uses pTi(τ | θ), while the inner expectation uses pTi+1(τ | φ).
A.1 MULTIPLE ADAPTATION GRADIENT STEPS
The multi-step adaptation variant constructs the adapted policy through several gradient steps, requiring intermediate environment interactions during training. At test time, the agent interacts only once and uses importance-weight correction.
- Multiple adaptation steps: The multi-step update constructs adapted parameters φ through M intermediate gradient steps.Intermediate policy parameters are denoted φm.
- Multiple adaptation steps: Each intermediate adaptation step requires environment interaction and sampling trajectories under the intermediate policy.The gradient must marginalize the intermediate policies and trajectories.
- Multiple adaptation steps: With intermediate transition distributions represented as delta functions, the multi-step objective retains the form of the single-step objective while integrating over intermediate trajectories.The gradient gains M additional terms based on those trajectories.
- Training and testing: Training uses rollouts from each intermediate policy without constraining interactions per environment, whereas testing uses one interaction and importance-weight correction.The training and testing interaction budgets therefore differ in the described procedure.
- Architectures: Policy and value-function architectures are illustrated in Fig. 9.The supplied figure passage identifies the figure as an architecture overview.
B ADDITIONAL DETAILS ON THE ARCHITECTURES
The experiments use memory-less MLP or recurrent LSTM policies and value functions, with meta-learned adaptation based on three gradient steps and adaptive step sizes.
- Architectures: MLP architectures are memory-less and reactive, whereas LSTM architectures use 64-unit embedding and recurrent layers.LSTM state is preserved within an episode and reset at each new episode.
- Architectures: The meta-learned adaptation methods are applied with both MLP and LSTM policies.The architecture comparison is part of the implementation details accompanying Fig. 9.
- Meta-updates: Meta-updates use 3 gradient steps with adaptive step sizes initialized at 0.001.The policy and value-function parameters are concatenated for θ and φ.
- Optimization details: Meta-gradient norms are clipped between -0.1 and 0.1 because early optimization steps often produced exploding updates.The outer optimization uses PPO while meta-updates use vanilla policy gradients on negative discounted rewards.
C.2 ON PPO AND ITS DISTRIBUTED IMPLEMENTATION
The RoboSumo experiments use distributed PPO, simultaneous training across varied environments, continuous observations and actions, and calibrated competitive agents with sparse and dense rewards.
- Large batch sizes were critical for learning in competitive RoboSumo, with 32,000 episodes per PPO epoch and batches of 8,000.The PPO clipping hyperparameter was ϵ = 0.2, and the KL penalty was 0.
- Distributed PPO versioned agent parameters and rollouts so multiple workers could generate experience in parallel for the optimizer.
- Training environments represented dysfunctional leg pairs in locomotion and different opponents in RoboSumo, with each mini-batch sampling from all environments.
- RoboSumo provides continuous observation and action spaces containing agent poses, joint states, velocities, and body-contact forces.Agents do not observe opponent velocities or limb positions directly, but can sense limbs through contact forces.
- RoboSumo combines sparse win, loss, and draw rewards with dense shaping rewards for pushing, approaching, hitting, and controlling movement.The winner receives 2000, the loser -2000, and draws give both agents -1000.
- Calibration trained agents with self-play, measured unadapted win rates, and increased weaker agents’ mass until matchups became balanced.The resulting before-and-after calibration win rates are reported in Table 2.
E ADDITIONAL DETAILS ON EXPERIMENTS
The experiments report average win rates over the final 25 rounds of 100-round iterated adaptation games, with 95% confidence intervals and LSTM base policies.
- Table 3 reports average win rates with 95% confidence intervals, using LSTM base policies and value functions with 64 hidden units.
E.2 TRUESKILL RANK OF THE TOP AGENTS
The top-agent analysis ranks MLP and LSTM populations by TrueSkill and estimates pairwise win rates, while also testing sensitivity to large opponent distributional shifts.
- The top-five MLP and LSTM agents are ranked by TrueSkill, with a heatmap showing a priori win rates among the top agents.
- TrueSkill models each agent’s skill as a normal distribution and supports estimating an a priori probability of one agent defeating another.
- 10 to 25% higher win likelihood was estimated for the best meta-learners against the best agents using other adaptation strategies within both policy families.
- Meta-learning became sensitive to large distributional shifts, achieving non-negative rewards only when the opponent changed by at most 4 self-play update steps per iteration.The tested opponent updates ranged from 1 to 10 steps per iteration.