Source-linked AI summary
Kickstarting Deep Reinforcement Learning
Simon Schmitt, Jonathan J. Hudson, Augustin Zidek, Simon Osindero, Carl Doersch, Wojciech M. Czarnecki, Joel Z. Leibo, Heinrich Kuttler, Andrew Zisserman, Karen Simonyan, S. M. Ali Eslami
TL;DR
Reinforcement-learning agents can require billions of environment steps, motivating methods that reuse previously trained agents to accelerate new training. Kickstarting combines policy distillation with population-based training while allowing teacher influence to adapt, and on DMLab-30 it substantially improves data efficiency, including matching a from-scratch agent in 9.58× fewer steps and surpassing its final performance by 42.2%.
Problem
Reinforcement-learning agents may require up to billions of environment steps, creating a need to train new agents faster using previously trained teachers.
Method
Kickstarting combines an auxiliary teacher-matching loss with reinforcement learning and automatically adjusts teacher influence over time without constraining teacher or student architectures.
Results
On DMLab-30, kickstarting achieves up to 1.5× single-teacher speedup, while multiple teachers match from-scratch performance in 9.58× fewer steps and surpass final performance by 42.2%.
Takeaways & Limitations
Kickstarting improves data efficiency and can shorten research iteration cycles in deep reinforcement learning while allowing students to surpass their teachers.
Abstract
from arXiv · showhide
We present a method for using previously-trained 'teacher' agents to kickstart the training of a new 'student' agent. To this end, we leverage ideas from policy distillation and population based training. Our method places no constraints on the architecture of the teacher or student agents, and it regulates itself to allow the students to surpass their teachers in performance. We show that, on a challenging and computationally-intensive multi-task benchmark (DMLab-30), kickstarted training improves the data efficiency of new agents, making it significantly easier to iterate on their design. We also show that the same kickstarting pipeline can allow a single student agent to leverage multiple 'expert' teachers which specialize on individual tasks. In this setting kickstarting yields surprisingly large gains, with the kickstarted agent matching the performance of an agent trained from scratch in almost 10x fewer steps, and surpassing its final performance by 42 percent. Kickstarting is conceptually simple and can easily be incorporated into reinforcement learning experiments.
1. Introduction
Kickstarting trains new reinforcement-learning students with previously trained teachers to reduce the cost of learning and iteration. The method combines policy distillation with population-based training, supports flexible teacher and student architectures, and shows strong gains on DMLab-30.
- Kickstarting uses previously trained agents as teachers so new students can learn more efficiently than agents trained from scratch.The approach addresses the high experience cost of increasingly complex reinforcement-learning tasks.
- The method combines policy distillation and population-based training without imposing architectural constraints on teachers or students.Its teacher influence is automatically adjusted during training rather than fixed throughout.
- 1.5× speedup is achieved with a single teacher over training a state-of-the-art agent from scratch on DMLab-30.
- 9.58× fewer steps are needed for a multiple-teacher student to match a from-scratch agent, while its final performance is surpassed by 42.2%.The multi-teacher scenario uses task-specific expert teachers.
2. Kickstarting RL Agents
Kickstarting trains a new student agent with pre-trained teachers while retaining the reinforcement-learning objective. It combines policy matching with reward maximisation, adapts teacher influence over time, and extends to multiple task-specific teachers.
- 2. Kickstarting RL Agents: Multiple teachers can specialize by task, allowing the student to learn from an appropriate expert in each task.Population based training can adjust each teacher’s λk schedule while also tuning learning rate and entropy regularisation.
- 2.1. Knowledge Transfer: The student maximises its own expected return rather than merely replicating the teacher’s behaviour.The method preserves the RL objective and allows the student to pursue higher rewards than the teacher.
- 2.1. Knowledge Transfer: Kickstarting combines an RL loss with a teacher–student cross-entropy term, weighted by λk on student-generated trajectories.The teacher supplies supervision while the student remains responsible for generating experience.
- 2.1. Knowledge Transfer: Unlike standard policy distillation, kickstarting retains reinforcement learning, uses student trajectories, and avoids forcing exact behavioural replication.The authors describe it as a continuous joint optimisation rather than a fixed two-phase distillation-then-fine-tuning procedure.
- 2.1. Knowledge Transfer: λk changes during training so supervision can provide an initially dense signal before optimisation shifts toward potentially sparse environmental rewards.The schedule can eventually set λk to zero, making the student independent of the teacher.
- 2.2. Kickstarting Actor-Critic: Kickstarting augments actor-critic learning while entropy regularisation encourages exploration beyond the teacher’s supervision.The framework is implemented in large-scale distributed settings such as IMPALA and uses on-policy actor-critic trajectories with value-based updates.
3. Related Work
Kickstarting builds on teacher–student distillation approaches in supervised and multitask reinforcement learning. It differs by combining student-generated experience with reward maximisation throughout training rather than treating distillation as a separate preliminary phase.
- 3. Related Work: Earlier approaches commonly train a student by supervised learning on experience gathered from expert policies.The related work includes model compression, policy distillation, and multitask reinforcement-learning methods.
- 3. Related Work: Kickstarting continuously combines distillation with reward maximisation from the beginning, avoiding arbitrary stopping criteria between learning phases.This distinguishes it from a two-phase procedure that first replicates teachers and later fine-tunes with rewards.
- 3. Related Work: Kickstarting differs from Distral because its focus is the student’s performance, whereas Distral uses the student as a communication channel while regularising teacher policies.The comparison identifies a different optimisation emphasis despite similarities in knowledge sharing.
4. Experimental Setup
Experiments use IMPALA agents with convolutional, recurrent, and policy/value components on the 30-task DMLab-30 benchmark. The setup compares small and large architectures and uses distributed population members with factorized teacher-distillation weights.
- Agent architectures: The agents use convolutional visual processing followed by an LSTM and linear policy and value heads, with an additional language LSTM when language input is available.
- Agent architectures: The small agent has 2 convolutional layers, while the large agent has 15 and represents the state-of-the-art architecture for these tasks.
- Distributed training: Each population member uses one high-capacity learner GPU worker and 150 environment-interacting actor workers, distributed evenly across the 30 tasks.
- Population-based training: Distillation weights are factorized so population-based training can strengthen or weaken all teacher influences simultaneously.
- Task suite: DMLab-30 contains 30 distinct tasks designed to challenge a broad range of agent cognitive capabilities.
- Evaluation: Performance is measured with mean capped human-normalised scores computed from returns over 30 million environment frames.
5. Experimental Results
Across DMLab-30 experiments, kickstarting improves training speed and final performance for single- and multi-teacher settings. Population-based control of the distillation weight supports student improvement beyond teachers, while reinforcement learning remains important for surpassing pure distillation.
- 5.1. Kickstarting With a Single Teacher: 6.92× speedup reaches score 30, while 1.53× reaches score 50 when a large student is kickstarted by a small teacher.The student reaches the small teacher’s final performance after about 1 billion frames versus the teacher’s 10 billion.
- 5.1. Kickstarting With a Single Teacher: +43.4% improvement lets the kickstarted student exceed its teacher, scoring 56.9 versus 39.7 on DMLab-30.The distillation weight falls to almost negligible levels after 2 billion steps, allowing continued independent learning.
- 5.1.1. PBT AND OTHER DISTILLATION WEIGHTING APPROACHES: PBT performs nearly as well as the best manually specified distillation-weight schedules and avoids sweeping over schedules.Constant weights can plateau near teacher performance, whereas quickly reducing linear schedules tend to work better.
- 5.2. Kickstarting With Multiple Teachers: 73.8 versus 56.9 DMLab-30 score shows multi-expert kickstarting outperforming a comparable single-teacher setup.The multi-expert student also performs far above an agent trained from scratch.
- 5.2.1. EXPERT KICKSTARTING VS. DISTILLATION: Pure policy distillation underperforms kickstarting, with the gap widening over training, indicating value in combining teacher guidance with the student’s own rewards.This comparison uses an equivalent architecture and removes typical reinforcement learning losses from the distillation-only agent.
- 5.2.2. ANALYSIS OF DISTILLATION WEIGHT λk: Separate per-teacher distillation schedules provide only a negligible boost, suggesting the multi-teacher procedure is not highly sensitive to this hyperparameter.The comparison includes fixed, shared-evolved, and separately evolved schedules.
6. Conclusion
Kickstarting shortens deep-RL research iteration cycles while enabling students to exceed both teachers and agents trained from scratch. The approach balances teacher advice with the student’s own learning objectives, suggesting a broader path toward agents built from prior knowledge.
- Kickstarting helps shorten deep-RL research iterations and enables students to outperform agents trained from scratch.
- Students can surpass their teachers because kickstarting balances environmental rewards with teacher advice rather than enforcing imitation alone.
- The authors propose kickstarting as a research direction for designing agents that absorb and use knowledge from previously trained agents.