PaperlayerView exact revision on arXiv

Source-linked AI summary

Introducing Symmetries to Black Box Meta Reinforcement Learning

Louis Kirsch, Sebastian Flennerhag, Hado van Hasselt, Abram Friesen, Junhyuk Oh, Yutian Chen

arXiv:2109.10781v2cs.LGcs.AIcs.NEstat.ML

TL;DR

Black-box meta-RL is flexible, but its generalisation to new environments is limited, while symmetries in backpropagation-based methods are not typically present in black-box systems. The paper adapts a symmetry-based supervised meta-learning approach to RL, and reports improved generalisation across unseen action and observation spaces, tasks, and environments.

Abstract

from arXiv · show

Meta reinforcement learning (RL) attempts to discover new RL algorithms automatically from environment interaction. In so-called black-box approaches, the policy and the learning algorithm are jointly represented by a single neural network. These methods are very flexible, but they tend to underperform in terms of generalisation to new, unseen environments. In this paper, we explore the role of symmetries in meta-generalisation. We show that a recent successful meta RL approach that meta-learns an objective for backpropagation-based learning exhibits certain symmetries (specifically the reuse of the learning rule, and invariance to input and output permutations) that are not present in typical black-box meta RL systems. We hypothesise that these symmetries can play an important role in meta-generalisation. Building off recent work in black-box supervised meta learning, we develop a black-box meta RL system that exhibits these same symmetries. We show through careful experimentation that incorporating these symmetries can lead to algorithms with a greater ability to generalise to unseen action & observation spaces, tasks, and environments.

Introduction

Meta-RL seeks algorithms that learn from interaction across environments, but black-box systems have not matched the generality of human-engineered algorithms. The paper hypothesizes that adding symmetries to black-box meta-learners can improve generalisation, and reports gains across several types of unseen settings.

Introduction

  • Black-box meta-RL jointly specifies the agent and learning algorithm in a neural network, but has not yet discovered general-purpose algorithms matching the generality of human-engineered ones.Backpropagation-based approaches are more prevalent, while black-box methods are expressive and may avoid some issues of backpropagation-based optimization.Source: Introduction · p. 1
  • The paper hypothesizes that adding symmetries to black-box meta-learners can improve generalisation to unseen environments.Source: Introduction · p. 1
  • SymLA reuses the same learned learning rule across network nodes, supports varying input, output, and architecture sizes, and is invariant to input and output permutations.Source: Introduction · p. 1
  • Experiments report improved generalisation with SymLA across changing action dimensions, permuted observations and actions, and substantially changed tasks and environments.Source: Introduction · p. 1

Preliminaries

The preliminaries define the RL objective and frame meta-RL as learning agents that adapt throughout their lifetimes across a distribution of environments. They contrast stationary policies with explicit update rules against black-box systems that directly learn a recurrent, history-dependent agent.

Reinforcement Learning

  • In the standard (PO)MDP formulation, an agent receives observations, selects actions, receives rewards, and seeks policy parameters maximizing expected discounted return.The environment dynamics specify next-state and reward probabilities; the episode length is T and the discount factor satisfies 0 < γ ≤ 1.Source: Reinforcement Learning · pp. 1–2
  • Figure 1 depicts SymLA's dense-network weights replaced by parameter-shared RNNs, with activations represented as forward and backward messages that can carry observations, prior actions, and rewards.Source: Figure 1 · p. 2

Meta Reinforcement Learning

  • Meta-RL seeks agents that learn throughout a multiepisode lifetime by using rewards to update behavior, maximizing expected return across a distribution of meta-training environments.Source: Meta Reinforcement Learning · p. 2
  • A lifetime-learning agent maps its prior state, observation, previous reward, and previous action to a new state and action; backpropagation-based systems separate policy from update rule, while black-box systems learn the recurrent map directly.For black-box RNNs, inputs include observation, reward, and action, and hidden state is not reset between episodes, allowing knowledge to accumulate over an agent's lifetime.Source: Meta Reinforcement Learning · p. 2

Symmetries in Meta RL

The section explains how symmetries in backpropagation-based learning support reuse across network elements and permutations, then contrasts black-box meta-RL's more specialized parameterization. That specialization can make black-box learners sensitive to permutations and tie their input and output sizes to their RNN parameters.

Symmetries in Meta RL

  • The paper frames differences between backpropagation-based and black-box learning dynamics as relevant to black-box methods' generalisation to novel environments.Source: Symmetries in Meta RL · p. 2

Symmetries in Backpropagation-based Meta RL

  • Backpropagation-based learning applies the same update rule across parameters, permits varying network sizes, and is invariant to input and output permutations when the objective respects those permutations.The objective's gradient must permute consistently, and the environment must accept the correspondingly permuted action; policy gradients are given as an example.Source: Symmetries in Backpropagation-based… · pp. 2–3

Insufficient Symmetries in Black-box Meta RL

  • Black-box systems assign unique meta-parameters to policy-state elements, encouraging element-specific updates and sensitivity to input and output permutations.Their input and output spaces must retain the same size because those spaces directly depend on the number of RNN parameters.Source: Insufficient Symmetries in Black-box… · p. 3
  • In the two-armed-bandit example, MetaRNN can favor the historically higher-paying first arm without observing rewards, a solution that performs optimally on the training distribution but fails to generalise to other distributions.The passage contrasts this fixed-policy bias with symmetric or REINFORCE-based learning, which cannot favor one arm absent observed rewards and instead learns from them.Source: Insufficient Symmetries in Black-box… · p. 3

Adding Symmetries to Black-box Meta RL

SymLA adapts variable shared meta learning to reinforcement learning by introducing shared learning rules and input/output permutation symmetries into a black-box meta-learner. Its design combines recurrent state updates with flexible input, output, and architecture sizes.

Adding Symmetries to Black-box Meta RL

Variable Shared Meta Learning

  • SymLA replaces each neural-network weight with an RNN whose shared parameters define message propagation and state updates.The RNNs exchange forward and backward messages; their states correspond to weights in a dense layer.Source: Variable Shared Meta Learning · pp. 3–4

RL Agent Inputs and Outputs

  • Observations enter the first RNN layer, previous actions enter the last, and the output message supplies unnormalized categorical-action logits.The experiments use discrete actions, with continuous control described as an adaptation.Source: RL Agent Inputs and Outputs · p. 4

Architecture Recurrence and Reward Signal

  • The single-layer architecture uses two levels of recurrence: within each RNN and through messages fed back into the layer.Each RNN also receives the previous reward, and the layer’s update is applied multiple times per environment step.Source: Architecture Recurrence and Reward… · p. 4

Symmetries in SymLA

  • By construction, SymLA shares learning rules across nodes, supports changes in input, output, and architecture sizes without changing meta-parameter count, and is invariant to input and output permutations.With i.i.d. initialized RNN states, permuted states can recover the original equations.Source: Symmetries in SymLA · p. 4

Learning / Inner Loop

  • In the inner loop, randomly initialized RNN states are updated as the agent interacts, and those states carry across environment resets during a lifetime.The method runs for L ≥ T steps when an environment episode has T steps.Source: Learning / Inner Loop · p. 5

Meta Learning / Outer Loop

  • The outer loop optimizes the agent’s lifetime reward with evolutionary strategies over sampled environments.The paper motivates evolutionary strategies as allowing long inner-loop horizons without backpropagation-related memory constraints.Source: Meta Learning / Outer Loop · p. 5

Experiments

Experiments compare SymLA with MetaRNN across bandits, permuted classic-control inputs and actions, reward-swapped tasks, and GridWorld-to-CartPole transfer. SymLA matches MetaRNN on similar bandits and shows generalisation in the tested unseen configurations where MetaRNN fails or over-specializes.

Experiments

  • The experiments test similar-environment learning, changing action counts, permuting observation and action order, switching task rewards, and transferring between environments.The study compares SymLA with MetaRNN across bandits, classic control, and grid worlds.Source: Experiments · p. 5

Learning to Learn on Similar Environments

  • On two-armed bandits with similar meta-training and meta-test settings, SymLA recovers performance similar to MetaRNN.The experiments report expected cumulative regret after 100 arm-pulls; lower regret is better.Source: Learning to Learn on Similar… · p. 5, Figure 3 · p. 5

Generalisation to Unseen Tasks

  • After reward assignments for the heart and trap are swapped, SymLA infers the new relationships while MetaRNN collects the wrong item.SymLA begins with near-zero rewards and improves through meta-test interactions.Source: Figure 6 · p. 6, Generalisation to Unseen Tasks · p. 6

Generalisation to Unseen Action Spaces

Generalisation to Unseen Observation Spaces

  • On permuted observations and actions in CartPole, Acrobot, and MountainCar, SymLA performance does not degrade, while MetaRNN fails on the untrained permutations.SymLA quickly learns the relevant observation and action ordering at meta-test time.Source: Generalisation to Unseen Observation… · p. 6, Figure 5 · p. 6

Generalisation to Unseen Environments

  • GridWorld-trained SymLA generalises to CartPole, whereas MetaRNN does not.The experiment uses random linear observation projections during meta-training to address the mismatch between binary GridWorld and continuous CartPole observations.Source: Figure 7 · p. 7, Generalisation to Unseen Environments · p. 7

Related Work

Related work includes black-box recurrent meta-RL, learned learning rules and fast weights, backpropagation-based meta-RL, and neural-network symmetries. The paper distinguishes its focus on symmetries in learning algorithms from symmetries used to reflect input-data structure.

Black-Box Meta RL

  • Black-box meta-RL uses policies with memory, such as RNN recurrence, to learn from reward signals, but these approaches lack the symmetries discussed here and tend to overfit.The cited examples include Schmidhuber (1993b), Hochreiter, Younger, and Conwell (2001), Wang et al. (2016), and Duan et al. (2016).Source: Related Work · p. 7

Learned Learning Rules & Fast Weights

  • Learned learning rules and fast weights use meta-learned mechanisms to update fast weights, often through outer products, and can be extended through parameter sharing.The paper notes that these approaches do not include all the symmetries it discusses.Source: Related Work · p. 7

Backpropagation-based Meta RL

  • Backpropagation-based meta-RL alternatives include learned initializations, gradient warping, hyper-parameter learning, and meta-learned objective functions.Source: Related Work · p. 7

Neural Network Symmetries

  • Prior neural-network symmetry work often reflects input structure, whereas this paper investigates the structure and symmetries of learning algorithms.The paper says the effects of symmetries in meta-learning had not been discussed in detail and experimentally investigates them in meta-RL.Source: Related Work · p. 7

Conclusion

The conclusion reports that SymLA incorporates symmetries identified in backpropagation-based meta-RL into a black-box meta-RL method. Its experiments show generalisation across altered action and observation spaces, unseen tasks, and different environments.

Conclusion

  • SymLA extends a black-box meta-learning method to meta-RL by incorporating symmetries found in backpropagation-based methods.Source: Conclusion · p. 7
  • The reported results include generalisation to varying arm counts, permuted observations and actions without performance degradation, and learning state-reward associations at meta-test time.Source: Conclusion · p. 7
  • The learned behavior transfers between grid world and unseen classic-control environments, and SymLA is described as less prone to overfitting than MetaRNNs.Source: Conclusion · p. 7

Bandits from Wang et al. (2016)

The bandit benchmarks vary how arm payout probabilities are sampled. They include independent uniform payouts and dependent payouts with differing difficulty ranges.

Bandits from Wang et al. (2016)

Hyper-parameters

The section specifies SymLA’s recurrent architecture, meta-optimization settings, and an observation transformation used for generalisation experiments.

SymLA Architecture

  • SymLA uses one recurrent layer with message size 8 in each direction, parameter-shared LSTMs of hidden size 16 (64 for bandits), and two micro ticks.The LSTMs produce the next state h_ab.Source: SymLA Architecture · p. 10

Meta Learning / Outer Loop

  • Evolution strategies estimate gradients with 10 evaluations per population sample (100 for bandits), followed by Adam; training uses population size 512 and task-specific settings.The learning rate and noise standard deviation are 0.01 and 0.035, respectively (0.2 each for bandits); inner-loop lengths are 500 (100 for bandits), with 4,000 outer steps for bandits and 20,000 otherwise.Source: Meta Learning / Outer Loop · p. 10

Generalisation to Unseen Environments

Scalability and complexity

The section characterizes SymLA’s computational and memory costs, its training-time comparison with MetaRNN, and the policy-update framing used in the comparison.

Scalability and complexity

  • SymLA’s inner-loop and meta-testing cost is O(N^2W) per environment step, while memory is independent of time steps and scales as O(N^2 + NW + MS).With evolution strategies, an outer step scales as O(N^2WLPE), where L is evaluated lifetime length, P population size, and E evaluations per particle.Source: Scalability and complexity · p. 10
  • At matched RNN count, SymLA is slower than MetaRNN by a factor of N^2 (N = 16 here), though its greater capacity may make fewer RNNs sufficient.Source: Scalability and complexity · p. 10
  • The comparison table covers fixed REINFORCE, backpropagation-based MAML, MetaGenRL, and LPG, black-box MetaRNN, and symmetry-equipped SymLA.Source: Table 1 · p. 11
  • The policy θ is stationary and updated by backpropagation at fixed intervals.Source: Scalability and complexity · p. 11

Code snippet

The code implements SymLA as recurrent message-passing layers embedded in an environment loop, with evolutionary-strategy gradient estimation and parameter updates.

Code snippet

  • Each layer tick concatenates forward and backward messages with auxiliary inputs, feeds them to an LSTM, and updates both message directions from the resulting outputs.Source: Code snippet · p. 11
  • The layer reads action logits from the forward messages and returns them alongside the updated LSTM and message state.Source: Code snippet · p. 11
  • The environment loop supplies flattened observations, reward, and termination signals with the previous action, samples an action from logits, then steps the environment.Source: Code snippet · p. 12
  • The code generates scaled noise for antithetic parameter variants, evaluates positive and negative variants in the environment, and computes an ES factor from the two losses using twice the noise standard deviation.Source: Code snippet · p. 12

Related papers

Loading 2109.10781v2…