PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Zero-shot Generalisation in Deep Reinforcement Learning

Robert Kirk, Amy Zhang, Edward Grefenstette, Tim Rocktäschel

arXiv:2111.09794v6cs.LGcs.AI

TL;DR

Zero-shot generalisation in RL seeks policies that transfer to diverse, changing, unseen deployment situations rather than overfitting training environments. This survey unifies terminology and formalism, categorises benchmarks and methods, and concludes that controllable variation is preferable to purely black-box procedural generation while several RL-specific settings remain underexplored.

  • Problem

    Real-world RL requires policies that remain robust and transfer or adapt across diverse, dynamic, and unseen environments.

  • Method

    The survey provides a common formalism and terminology for ZSG problems, then categorises benchmarks, evaluation protocols, and methods across the field.

  • Results

    The survey concludes that purely black-box procedural content generation is unsuitable for testing specific generalisation forms, recommending controllable variation alongside procedural generation.

  • Takeaways & Limitations

    Future ZSG research should examine online adaptation and RL-specific problems, and develop benchmarks for underexplored settings such as offline RL and reward-function variation.

  • Takeaways & Limitations

    Understanding RL generalisation remains at an early stage, and purely procedural-content-generation environments limit tight control over environmental variation for scientific experimentation.

Abstract

from arXiv · show

The study of zero-shot generalisation (ZSG) in deep Reinforcement Learning (RL) aims to produce RL algorithms whose policies generalise well to novel unseen situations at deployment time, avoiding overfitting to their training environments. Tackling this is vital if we are to deploy reinforcement learning algorithms in real world scenarios, where the environment will be diverse, dynamic and unpredictable. This survey is an overview of this nascent field. We rely on a unifying formalism and terminology for discussing different ZSG problems, building upon previous works. We go on to categorise existing benchmarks for ZSG, as well as current methods for tackling these problems. Finally, we provide a critical discussion of the current state of the field, including recommendations for future work. Among other conclusions, we argue that taking a purely procedural content generation approach to benchmark design is not conducive to progress in ZSG, we suggest fast online adaptation and tackling RL-specific problems as some areas for future work on methods for ZSG, and we recommend building benchmarks in underexplored problem settings such as offline RL ZSG and reward-function variation.

1. Introduction

Zero-shot generalisation addresses the gap between RL policies trained in fixed environments and deployment in changing, unseen environments. The survey unifies this problem class, organises its benchmarks and methods, and identifies priorities for more precise and useful research.

  • Real-world RL requires policies that transfer and adapt to unseen environments because deployment settings are dynamic and unpredictable.
  • Many standard RL benchmarks evaluate policies on the same environment used for training, unlike deployment scenarios with different environment instances.
  • The survey formalises ZSG as a broad class of problems, distinguishing generalisation types such as combinatorial, interpolation versus extrapolation, and single-factor versus multi-factor variation.
  • The review focuses on zero-shot evaluation without additional training or test-environment data, and covers single-agent rather than multi-agent RL.
  • The survey categorises ZSG benchmarks by environments and evaluation protocols, arguing that purely procedural content generation limits precise experimentation.
  • The survey categorises methods and highlights fast online adaptation, RL-specific issues, novel architectures, model-based RL, and environment generation as future directions.
  • The survey recommends studying underexplored settings including offline RL generalisation, reward-function variation, context efficiency, and continual RL.

2. Related Work: Surveys In Reinforcement Learning Subfields

Existing surveys cover related areas such as continual, robust, sim-to-real, transfer, multi-task, exploration, and curriculum RL, but not ZSG explicitly. The paper positions ZSG as a broader class requiring a shared formalism for comparing its subproblems.

  • Previous surveys address related RL subfields, but none explicitly focuses on zero-shot generalisation in RL.
  • Continual RL and transfer RL involve training and deployment differences, but generally allow extra training or adaptation in the target environment, unlike ZSG.
  • Robust RL studies a specific form of environment-model misspecification through worst-case optimisation, making it a subproblem within the broader generalisation class.
  • The field needs a formalism that enables researchers to specify and compare different ZSG problems rather than treating ZSG as a single setting.

3. Formalising Zero-shot Generalisation In Reinforcement Learning

The survey formalises zero-shot generalisation (ZSG) in RL as a broad class of deployment problems involving policies trained on some environment instances and evaluated without additional test-environment training. It uses this framework to clarify evaluation, structural assumptions, and research priorities.

  • 3. Formalising Zero-shot Generalisation In Reinforcement Learning: ZSG is formalised as a class of problems, so research must specify which generalisation problems it targets.The framework also distinguishes subproblems through assumptions about structure and context.
  • 3.2 Background: Reinforcement Learning: The RL background is grounded in MDPs and POMDPs, with states, actions, rewards, transitions, observations, and initial-state distributions.These formalisms provide the underlying language for describing contextual RL environments.
  • 3.3 Contextual Markov Decision Processes: In this framework, context remains fixed within an episode and changes only between episodes, representing a seed, ID, or parameter vector that selects an instance.Training and testing collections are determined by different context distributions.
  • 3.3 Contextual Markov Decision Processes: A contextual MDP represents a collection of task instances whose context determines the reward, transition, initial-state, and emission functions while leaving the action space fixed.A fixed context yields one context-MDP, such as a particular task or environment instance.
  • 3.4 Training And Testing Contexts: The survey does not assume shared structure between context-MDPs, because useful assumptions depend on the specific ZSG problem.Additional structure may be needed for learning, but no single assumption unifies all ZSG problems.
  • 3.4 Training And Testing Contexts: The generalisation gap can complement test performance, but zero gap may reflect poor performance and incomparable reward scales can make its magnitude uninformative.The survey therefore recommends problem-specific benchmarks and overall performance, using the gap as an auxiliary metric.
  • 3.6 Additional Assumptions For More Feasible Generalisation: Evaluating on only one testing context can produce a policy overfit to that context rather than one that generalises across similar contexts.The testing context set should support the broader robustness objective.
  • 3.7 Remarks And Discussion: Real-world deployment motivates offline or simulation training followed by zero-shot use, because current RL algorithms are not yet safe or sample-efficient enough for online learning.A single model can avoid repeatedly paying verification and auditing costs after updates.

4. Benchmarks For Zero-shot Generalisation In Reinforcement Learning

The survey catalogues ZSG environments and evaluation protocols, showing broad variation in benchmark styles but important limitations in purely PCG-based designs. It advocates combining procedural generation with researcher-controlled factors to support targeted, scientifically interpretable tests.

  • Environment categories: PCG environments use random seeds as context-set inputs, whereas controllable environments directly expose factors of variation in context-MDPs.The survey treats black-box PCG as an implicit combination of discrete and continuous factor spaces whose values are selected by random generation.
  • Environment trends: State variation appears in 42 environments (76%), while observation, reward, and dynamics variation appear in 29 (53%), 20 (36%), and 19 (35%), respectively.Most environments combine multiple variation types; none has solely reward variation, and only one has solely dynamics variation.
  • Evaluation protocols: Evaluation protocols specify training and testing context sets, sampling restrictions during training, and the number of permitted samples.For controllable environments, protocols can test interpolation, extrapolation, and combinatorial interpolation by selecting factor ranges and combinations.
  • Evaluation protocols: Protocol C trains on nearly the full PCG context set and tests on held-out seeds, primarily measuring robust optimisation rather than zero-shot generalisation beyond memorisation.Protocol A uses one training context, while Protocol B uses a small sampled training set and tests across the full space.
  • Benchmark design: Purely PCG environments limit evaluation to training-set size and make targeted factor-specific generalisation difficult because generated contexts lack structured, controllable variation.This makes isolating specific bottlenecks and measuring progress on particular ZSG problems difficult.
  • Benchmark design: The survey recommends combining PCG with fine-grained researcher control, preserving environmental variety while enabling precise experiments on disentangled sources of generalisation.Obstacle Tower and MiniHack illustrate this compromise through procedural components alongside controllable visual, layout, or full-MDP specifications.
  • Compositional generalisation: Structured controllable context spaces enable systematicity tests through unseen factor combinations and productivity tests through values beyond those seen during training.These forms correspond to combinatorial interpolation and ordinal or continuous extrapolation, respectively.

5. Methods For Zero-shot Generalisation In Reinforcement Learning

The survey organises ZSG methods according to whether they reduce train-test mismatch, explicitly handle environmental differences, or address RL-specific optimisation issues. This taxonomy supports comprehensive comparison and reveals underexplored areas for future research.

  • Method taxonomy: ZSG methods address settings where training and testing context sets differ, encompassing multiple problem types and corresponding method styles.
  • Method taxonomy: The taxonomy groups methods that increase training-testing similarity, explicitly handle environment differences, or target RL-specific issues and optimisation improvements.
  • Research gaps: The comprehensive classification exposes under-explored areas in ZSG research and informs future method development.

5.1 Increasing Similarity Between Training And Testing

Methods in this category make training contexts or data more similar to expected test contexts through data augmentation, domain randomisation, and environment generation. These approaches can improve zero-shot performance but face sample-efficiency, variation-coverage, and environment-generation constraints.

  • Data Augmentation and Domain Randomisation: Data augmentation adds transformed observations or enforces invariance, while domain randomisation samples training environments from varied parametrisations.DA may adjust training data or regularise representations; DR creates and samples a non-singleton training context set.
  • Data Augmentation and Domain Randomisation: DA methods improve performance on CoinRun, OpenAI Procgen, and some visually distracting DeepMind Control variants.Reported methods include UCB-DrAC, Mixreg, PAADA, RandFM, MixStyle, and SODA.
  • Data Augmentation and Domain Randomisation: Augmentation timing matters: some transformations help during training, whereas others primarily regularise the final policy.SECANT combines weak-augmentation expert training with policy distillation because strong augmentations can hinder performance.
  • Data Augmentation and Domain Randomisation: Increasing augmentation or randomisation makes optimisation harder and often reduces sample efficiency.Robust visual encoders trained with supervised or self-supervised methods offer an alternative to heavily randomised policy training.
  • Environment Generation: Environment generation learns useful, solvable context distributions when randomly sampling all possible contexts would include impossible or trivial training instances.Current methods require fine-grained context generation and primarily target state-space variation.
  • Environment Generation: PAIRED outperforms standard domain randomisation and a POET-like method on zero-shot generalisation to unseen grid-world levels.The cited comparison evaluates zero-shot performance on unseen levels.

5.2 Handling Differences Between Training And Testing

These methods address train-test differences by encoding assumptions about invariant or relevant features, applying regularisation, learning invariant representations across contexts, or meta-learning adaptation. Their shared strategy is to shape policies or representations toward variation expected to persist across environments.

  • Encoding Inductive Biases: Large pretrained language models improve performance on novel instructions by encoding domain knowledge for reward-function variation.SHIFTT and TransferLanfLfP target instruction-following tasks with novel goal specifications.
  • Encoding Inductive Biases: Inductive-bias methods alter architectures or representations so policies ignore irrelevant changing features or process structured variation appropriately.Examples include adversarial invariance, disentanglement, relational architectures, selective attention, and planning modules.
  • Standard Regularisation: Standard regularisation encourages simpler policies through mechanisms such as L2 weight decay, dropout, and information bottlenecks.Information-bottleneck methods report improved performance on Procgen, random mazes, and continuous-control tasks.
  • Standard Regularisation: All evaluated CoinRun regularisation techniques improve performance separately, while combining them improves performance further with minimal gain beyond individual methods.The evaluated techniques include data augmentation, dropout, batch normalisation, L2 weight decay, policy entropy, and their combination.
  • Learning Invariance Across Contexts: Multiple training contexts support learning invariant representations, including through causal inference, bisimulation or policy-similarity metrics, and invariant-risk approaches.These methods assume that features or representations stable across training contexts will generalise to test contexts.
  • Meta-learning Adaptation: Meta-learning methods such as RL2 learn adaptation functions during training, allowing recurrent hidden states to adapt across episodes without resetting at boundaries.RL2 can often adapt within a single episode.

5.3 RL-Specific Problems And Improvements

RL-specific methods address optimisation difficulties beyond supervised-learning generalisation, especially non-stationary data and exploration. Better policy-value optimisation and model-based learning can improve zero-shot benchmark performance without directly targeting ZSG.

  • RL-Specific Optimisation: RL optimisation adds non-stationary data and exploration challenges that interact with zero-shot generalisation.The survey distinguishes methods targeting these RL-specific issues from methods improving training without overfitting.
  • RL-Specific Optimisation: ITER addresses non-stationarity by iteratively distilling the current policy network because RL training can learn non-generalising features despite equal training performance.The passage identifies policy feature learning as the mechanism affected by non-stationary training.
  • Better Optimisation without Overfitting: PPG and DAAC improve Procgen performance by separating policy and value optimisation while distilling value or advantage information into the policy.DAAC distils a GAE advantage function, reducing reliance on details predictive of value but not optimal action selection.
  • Model-Based RL: MuZero Reanalyse achieves much-improved OpenAI Procgen performance over state-of-the-art model-free methods at much lower sample complexity.The result supports model-based RL as a potential route to improved zero-shot generalisation.
  • Scope of Included Methods: Methods not designed specifically for ZSG are included when they improve test-time performance on zero-shot generalisation benchmarks.The survey suggests benchmarks such as Procgen could become standard RL benchmarks rather than being treated as ZSG-specific.

5.4 Discussion

The discussion argues for classifications that expose problem structure, clearer treatment of inductive biases, and greater attention to RL-specific forms of generalisation. It identifies non-visual variation, including dynamics, state, and reward, as less studied despite its importance.

  • Method Categorisation: Tables 2 and 3 categorise methods by evaluated variation and by their main adjustment to the training environment, architecture, or objective.Green, red monospace, and blue italic styles identify environment, architecture, and objective or loss adjustments, respectively.
  • Alternative Classifications: The survey’s classification balances implementation differences with problem motivations and solution approaches, helping identify methods and underexplored research areas.A benchmark-only classification may obscure which task differences matter and which methods are likely to transfer.
  • Strong Generalisation Requires Inductive Biases: Hard combinatorial interpolation and extrapolation problems require explicit inductive biases when deployment contexts cannot be generated or training contexts are limited.Researchers should analyse how a bias helps and when it may hinder extrapolation in other settings.
  • Going Beyond Supervised Learning as Inspiration: Supervised-learning methods have especially influenced visual generalisation, whereas dynamics, state, and reward variation remain less studied.These non-visual challenges are described as equally important and specific to RL.

6. Discussion And Future Work

The discussion argues that useful RL generalisation must address realistic variation, limited contexts, offline data, and changing environments. It recommends controllable benchmarks and methods targeting underexplored, RL-specific challenges.

  • Generalisation beyond zero-shot policy transfer: Zero-shot policy transfer is useful, but continual RL may better represent environments that change over time.The authors recommend moving beyond strict zero-shot assumptions when deployment requires continual adaptation.
  • Real-world evaluation: Future benchmarks should measure context efficiency by evaluating methods with only a few training contexts, alongside sample efficiency.The survey proposes studying performance across different training-context-set sizes and investigating trade-offs between sample and context costs.
  • Real-world evaluation: Offline RL and ZSG should be combined by training on offline data and evaluating policies on unseen contexts.A tractable intermediate setting is offline pretraining followed by short online fine-tuning, with generalisation evaluated afterward.
  • Benchmark design: New benchmarks should combine procedural generation with controllable variation factors to support both diversity and precise experimentation.Controllable context sets also enable multidimensional evaluation and more targeted study of generalisation.
  • Underexplored variation: Reward-function variation is difficult but necessary for policies that perform multiple tasks and generalise to unseen tasks without further training data.The Principle of Unchanged Optimality is often difficult or impossible to verify in this setting.
  • Methods: Important future methods must address RL-specific factors such as non-stationary data, bootstrapping, TD learning, and exploration.The survey also identifies better context sampling and stronger methods for challenging, realistic variation as useful directions.

7. Conclusion

The conclusion presents ZSG as an important but still developing framework for making RL more applicable to real-world problems. It recommends broader evaluation, stronger benchmark control, and methods suited to stronger forms of variation.

  • Key contributions: The survey unifies terminology and formalism for ZSG, and categorises its benchmarks and existing methods.Its benchmark taxonomy separates environments from evaluation protocols.
  • Problem setting: Zero-shot policy transfer is useful as a foundation, but future work should also study continual RL.Continual RL can relax restrictions associated with strict zero-shot transfer.
  • Benchmarks: Purely black-box PCG environments are poorly suited to testing specific generalisation forms; combining PCG with controllable variation is recommended.This combination supports both environmental variety and scientific experimentation.
  • Evaluation: Real-world evaluation should consider both sample efficiency and context efficiency, including performance across training-context-set sizes.This metric provides more information for choosing between methods.
  • Future directions: Offline-RL generalisation remains underexplored, while dynamics and reward-function variation receive limited attention relative to observation and state-space variation.The conclusion identifies these stronger forms of variation as priorities for future research.
  • Methods: Stronger ZSG problems require stronger inductive biases, whose intended scope and benchmark relevance should be stated explicitly.The survey also identifies model-based RL, new architectures, fast online adaptation, and RL-specific problems as future method directions.
  • Impact: The survey aims to clarify the field and stimulate research useful to researchers and practitioners inside and outside RL.It presents itself as a reference point for ongoing work on ZSG.

Appendix A. Other Structural Assumptions on MDPs

The appendix describes structured MDP assumptions that factor states, transitions, rewards, objects, or observations. These structures can make systematic generalisation to unseen combinations more explicit or tractable.

  • Factored MDPs: Factored MDPs represent the state space as discrete variables and impose conditional independence structure on transitions.Each factor can depend on a subset of parent factors from the previous time step.
  • Factored MDPs: Factored reward functions decompose rewards across state factors, with R_i(s_i, a) representing the reward associated with factor S_i.This extends the factorised representation beyond transition dynamics.
  • Systematic generalisation: Factored MDPs can encode contexts as combinations of factors, enabling systematic generalisation to unseen combinations of previously observed factors.Training may cover factor values while testing uses combinations not encountered during training.
  • Other structured MDPs: Relational MDPs describe object types, fluent schemata, action schemata, domain objects, transitions, and rewards.The representation is based on structured relations among objects.
  • Other structured MDPs: Object-oriented MDPs simplify relational MDPs by using Boolean transition terms over relations and object attributes.Despite being less constrained, they remain significantly more constrained than contextual MDPs and can be difficult for complex systems.
  • Other structured MDPs: Block MDPs separate a finite unobservable state space from an observable, potentially infinite observation space through latent transitions and emissions.They are useful when rich observations correspond to a much smaller underlying state space.
Loading 2111.09794v6…