Mix&Match - Agent Curricula for Reinforcement Learning

1 Introduction

• Complex action spaces and challenging training regimes increasingly require more powerful models, making effective training difficult.Source: §1 · p. 1
• Mix&Match progressively trains a final agent through structurally different intermediate agents, ordered by training complexity rather than architectural complexity.The approach changes how policies are represented internally instead of modifying the underlying task specification.Source: §1 · p. 2
• Unlike task-oriented curricula, agent curricula apply when researchers cannot modify the task or environment specification.The two curriculum types can also be combined where task modification is possible.Source: §1 · p. 2

Mix & Match: An overview

• M&M combines multiple agents as a training-time mixture, transferring information through shared experience, shared architectural elements, and distillation-like KL matching.The mixture weights are adapted until only the most complex target agent remains active.Source: Mix & Match: An overview · p. 2

2 Related work

• The work extends automated or mixed-difficulty task curricula by applying the curriculum idea to training agents with different models.This reframes the curriculum elements as models rather than tasks.Source: §2 · p. 2
• Net2Net and related methods grow model capacity while preserving functions, forming an implicit architecture curriculum through successive model transformations.Source: §2 · p. 2
• Knowledge distillation transfers functional behavior across differently structured models and has been used to compress or coordinate multiple policies.Source: §2 · p. 2
• M&M unifies these threads around online, end-to-end training of model curricula from scratch.Source: §2 · p. 2

3 Method details

• The mixture policy selects among agents with increasing learning complexity, while the final objective targets the most complex policy.Agents may share parameters, and the curriculum can contain an arbitrary number of policies.Source: §3 · pp. 2–3

3.1 Policy mixing

• The mixture weights begin with the simpler policy and are adapted toward the target policy to control long-term learning performance.The authors use a categorical selector or an explicitly mixed policy; experiments use explicit mixing for variance reduction.Source: §3 · p. 3, §3.1 · p. 3

3.2 Knowledge transfer

• A distillation-like cost aligns policies because mixing alone can make constituent experts compete rather than share knowledge.The transfer loss is applied using trajectories sampled from the control mixture, and its weight disappears when switching to the target policy.Source: §3 · p. 3, §3.2 · p. 3
• With a memory buffer, the target policy can converge to the simpler policy under sufficient capacity, experience, and reasonable assumptions.Source: §3.2 · p. 3

3.3 Adjusting α through training

• Hand-designed schedules require problem-specific fitting and can miss useful non-monotonic changes in mixture weights.Source: §3.3 · p. 4

3.4 Population based training and M&M

• Population-based training adapts mixture weights online by training agents in parallel, copying stronger agents, and perturbing hyperparameters.Evaluation can use mixed-policy reward or directly assess the final target policy, depending on the problem.Source: §3.3 · p. 4, §3.4 · p. 4

4 Experiments

• The experiments use batched off-policy actor-critic training in DeepMind Lab and evaluate action-space, architectural, and multitask curricula with adaptive mixing coefficients.The final experiments use two agents in each curriculum, while mixing can generate intermediate policies between them.Source: §4 · pp. 4–5, Figure 2 · p. 5

4.1 Curricula over number of actions used

• M&M applied to a 756-action space learns quickly and surpasses the final performance of the 9-action baseline and the standalone big-action-space agent.The small action space accelerates early learning but limits final performance, whereas M&M combines fast learning with higher final performance.Source: §4.1 · p. 5, Figure 3 · p. 6, Figure 4 · p. 6
• All M&M action-space variants outperform the baselines in data efficiency and performance, while shared-head and masked-KL variants are comparable to or worse than the original formulation.The masked-KL variant may be more useful for non-factorised complex action spaces because it can provide the only signal encouraging exploration of new actions.Source: §4.1 · pp. 5–6, Figure 3 · p. 6
• The action-space curriculum switches fully to the big action space early, while new actions become unlocked and the final action distribution becomes more entropic.This indicates that the small action space mainly supports the initial learning phase before the agent relies on the larger space.Source: §4.1 · p. 6, Figure 5 · p. 6

4.2 Curricula over agent architecture

• Transitioning from a feedforward core to an LSTM increases final human-normalised performance by 20% across the tasks, although it is slower than the feedforward baseline.The larger FF+LSTM skip-connection model also improves performance but usually learns more slowly, and its size may explain an outlier on the LT task.Source: §4.2 · pp. 6–7, Figure 6 · p. 7, Figure 7 · p. 7
• An M&M-like model with two LSTM agents underperforms the other baselines in speed and performance, suggesting that the gains require the proposed agent transition rather than KL regularisation alone.The comparison supports the role of the full Mix&Match method, not merely its distillation cost.Source: §4.2 · p. 7
• PBT finds curricula of different lengths: navigation tasks switch quickly toward the LSTM, whereas lasertag delays the switch and a fixed-maze task can move back and forth through a mixture.Unconstrained adaptation of α permits these task-dependent reversals during training.Source: §4.2 · p. 7, Figure 8 · p. 7

4.3 Curricula for multitask

• In multitask learning, M&M produces more uniform performance across tasks, improving the challenging lasertag environments while slightly sacrificing performance on navigation.Policy-space knowledge transfer is invariant to reward scaling and yields a higher minimum score than the ordinary multitask solution.Source: §4.3 · pp. 7–8, Figure 9 · p. 8

5 Conclusions

• Mix & Match improves final performance and accelerates learning for complex agents in challenging environments.The framework uses a mixture policy, shared information, and distillation-like KL matching while ultimately selecting the most complex target agent.Source: §5 · p. 8
• The method trains agents through an implicit curriculum over agents of different training complexities rather than task variants.Component weightings adapt over time until only the complex target agent remains active.Source: §5 · p. 8
• M&M functions as a wrapper compatible with existing agent architectures and on- or off-policy reinforcement-learning schemes.The authors also note that its principles can extend beyond policy-based agents to value-based approaches such as Q-learning.Source: §5 · p. 8
• Across the presented applications, M&M trains complex agents more effectively and in less time than training them without the framework.The authors present these applications as support for the generality of the approach.Source: §5 · p. 8

A Network architectures

• The default architecture uses convolutional layers with 16 8x8 stride-4 kernels and 32 4x4 stride-2 kernels.Source: A Network architectures · p. 9
• A 256-neuron linear layer is concatenated with one-hot encoded representations of the last action and last reward.Source: A Network architectures · p. 9
• The architecture includes an LSTM core with 256 hidden units.Source: A Network architectures · p. 9
• Separate linear layers project the LSTM representation to policy logits followed by softmax and to the baseline.Source: A Network architectures · p. 9
• Depending on the experiment, some architectural elements are shared or replaced.Source: A Network architectures · p. 9

B PBT (Jaderberg et al., 2017a) details

• PBT controls α, learning rate, and entropy-cost regularisation across populations of 10 agents.Source: B PBT (Jaderberg et al., 2017a) details · p. 9
• Learning rate and entropy regularisation are randomly multiplied by 1.2 or 0.8 during exploration.Exploration occurs independently with probability 25% whenever a worker is ready.Source: B PBT (Jaderberg et al., 2017a) details · p. 9
• The explore operator for learning rate and entropy regularisation randomly multiplies the corresponding value by 1.2 or 0.8.Source: B PBT (Jaderberg et al., 2017a) details · p. 9
• The exploit operator copies weights and hyperparameters from a randomly selected agent when its performance is significantly better.Source: B PBT (Jaderberg et al., 2017a) details · p. 9
• Workers become eligible for adaptation every 300 episodes, with significance assessed using a 5% p-value threshold over averaged returns from the last 30 episodes.Source: B PBT (Jaderberg et al., 2017a) details · p. 9
• Initial hyperparameters use log-uniform ranges of 1e-5–1e-3 for learning rate, 1e-4–1e-2 for entropy cost, and 1e-3–1e-2 for α.Source: B PBT (Jaderberg et al., 2017a) details · p. 9

B.1 Single task experiments

• The evaluation function uses πmm rewards.Source: §B.1 · p. 9

• Each learner requires a separate evaluation worker because evaluation uses mixed-policy rewards.Source: §B.2 · p. 9

C M&M details

• Knowledge transfer uses λ=1.0 for action-space experiments and λ=100.0 for agent-core and multitask experiments, with gradients propagated through both policies.The teacher is regularised toward the student to limit divergence during training.Source: C M&M details · p. 9

D IMPALA (Espeholt et al., 2018) details

• Training uses 100 CPU actors per learner, one K80 GPU per learner, V-trace correction, 100-step unrolls, RMSProp, discounting 0.99, clipped rewards, and action repeat 4.Source: D IMPALA (Espeholt et al., 2018) details · p. 10

E Environments

• DM Lab provides 96 × 72 × 3 RGB observations at 60 fps.Source: E Environments · p. 10

E.1 Explore Object Locations Small

• Explore Object Locations Small uses procedurally generated mazes with changing layouts, apple locations, and visual themes; collecting all apples resets the environment.Each apple gives 1 point.Source: §E.1 · p. 10

E.2 Nav Maze Static 01/02

• Nav Maze Static 01 fixes maze geometry, theme, and object positions while randomising the spawn location; apples are worth 1 point and a calabash 10 points.Collecting the calabash resets the environment.Source: §E.2 · p. 10
• Nav Maze Static 02 differs from Static 01 by being significantly larger.Source: §E.2 · p. 10

E.3 LaserTag Horseshoe Color

• LaserTag Horseshoe Color places six bots and collectible Orb and Disc Gadgets in a wide horseshoe-shaped room.Source: §E.3 · p. 10

E.4 LaserTag Chasm

• LaserTag Chasm requires jumping for pickups above a chasm, where falling costs 1 point, against four built-in bots.Source: §E.4 · p. 10

• With N→∞ trajectories from a fixed mixture policy and a sufficiently large softmax student, minimising Eq. 1 converges to π1 under a globally convergent optimiser.Source: F Proofs · p. 10
• Because the softmax student assigns non-zero probability to every action, the mixture policy also covers every action in every state.Source: F Proofs · p. 10
• The mixture-generated dataset may replicate states in different proportions from π1, but a sufficiently large network can still minimise the loss over the finite dataset.Source: F Proofs · p. 10
• Global convergence is guaranteed for linear teacher and student policies and is hypothesised for sufficiently large, well-initialised deep networks.Source: F Proofs · p. 10

G On α based scaling of knowledge transfer loss

• The α-dependent factor makes the transfer KL gradually disappear as M&M switches to the final agent, equivalently making divergence from the target vanish at switching.Source: G On α based scaling of knowledge… · p. 11
• The two losses share a cross-entropy term and differ in their entropy regularisation.Source: G On α based scaling of knowledge… · p. 11
• The proposed loss is an upper bound on the mixed-policy KL objective because it is nearly equivalent but applies a more severe regularisation factor.Source: G On α based scaling of knowledge… · p. 11
• Preliminary experiments found both cost formulations similarly effective, so further research is needed to quantify their differences.Source: G On α based scaling of knowledge… · p. 11

H On knowledge transfer loss

• Bidirectional KL transfer pulls the student toward the teacher while regularising the teacher toward the student and uniformity, discouraging teacher overtake and favouring switching.Source: H On knowledge transfer loss · p. 11
• The paper does not evaluate deep distillation methods such as activation or Jacobian matching, although they may be needed for much larger models and more complex domains.Source: H On knowledge transfer loss · p. 11