PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Policy Optimization with Generalist-Specialist Learning

Zhiwei Jia, Xuanlin Li, Zhan Ling, Shuang Liu, Yiran Wu, Hao Su

arXiv:2206.12984v1cs.LG

TL;DR

Training across diverse environment variations can be inefficient, and generalist policies may plateau even though specialists can master smaller subsets. GSL trains a generalist first, then specialists whose demonstrations guide continued generalist learning; the paper reports improved performance on Procgen training and unseen variations and a broader ability to escape plateaus.

  • Problem

    Training across many diverse environment variations is challenging: RL methods can require very large sample counts while still achieving poor performance.

    Source: §1 · p. 1

  • Method

    GSL trains a generalist until its performance plateaus, launches specialists from its checkpoint on subsets of variations, and uses their demonstrations as auxiliary rewards for continued generalist training.

    Source: §1 · p. 2, §5 · p. 4

  • Results

    GSL can help agents escape suboptimal plateaus and reach high reward; on Procgen, it improves final performance over PPO/PPG on both training and unseen test variations.

    Source: §5.5 · p. 8, §7 · p. 9

  • Takeaways & Limitations

    The framework is a meta-algorithm that combines an RL algorithm with learning from demonstrations to address training across many environment variations.

    Source: §5 · p. 4, §7 · p. 9

    The plateau criterion requires some hyperparameter tuning: starting specialists too early loses initial-learning efficiency, while starting too late prolongs the plateau and can make fine-tuning harder.

    Source: §6.3 · p. 9

Abstract

from arXiv · show

Generalization in deep reinforcement learning over unseen environment variations usually requires policy learning over a large set of diverse training variations. We empirically observe that an agent trained on many variations (a generalist) tends to learn faster at the beginning, yet its performance plateaus at a less optimal level for a long time. In contrast, an agent trained only on a few variations (a specialist) can often achieve high returns under a limited computational budget. To have the best of both worlds, we propose a novel generalist-specialist training framework. Specifically, we first train a generalist on all environment variations; when it fails to improve, we launch a large population of specialists with weights cloned from the generalist, each trained to master a selected small subset of variations. We finally resume the training of the generalist with auxiliary rewards induced by demonstrations of all specialists. In particular, we investigate the timing to start specialist training and compare strategies to learn generalists with assistance from specialists. We show that this framework pushes the envelope of policy learning on several challenging and popular benchmarks including Procgen, Meta-World and ManiSkill.

1. Introduction

Training across many environment variations can make reinforcement learning inefficient, while a generalist may plateau as variation grows. GSL combines a generalist’s faster early learning with specialists’ high returns by training specialists after the plateau and using their demonstrations to guide the generalist.

1. Introduction

  • Many-variation training can remain inefficient: PPO on 1,000 Procgen levels may perform poorly after hundreds of millions of samples, while visual navigation across about 100 scenes may require billions.Prior approaches include automatic curricula, representation constraints, and decoupling policy and value learning.Source: §1 · p. 1
  • A generalist is one policy for all variations, whereas a specialist masters only a subset; the goal is to use specialist experience to aid generalist optimization.Source: §1 · p. 1
  • Shared early trajectory stages can favor joint generalist learning, but diverse later states can strain prediction and cause forgetting or ignorance; specialists on small subsets can attain higher returns there.The paper links these challenges to performance plateaus.Source: §1 · pp. 1–2
  • GSL trains a generalist on all variations until progress stalls, then trains specialists cloned from its checkpoint on small subsets and uses their demonstrations as auxiliary rewards.The paper evaluates the framework on subsets of Procgen and ManiSkill.Source: §1 · p. 2

2. Related Work

Related work covers divide-and-conquer learning, policy distillation, large-scale RL, and multi-task RL. GSL differs in when it starts specialist training and in its applicability to variations without semantic task encodings.

Divide-and-Conquer in RL

  • Prior divide-and-conquer methods split state spaces, train local policies, and merge them through imitation; GSL emphasizes when to start specialists and scales to many specialists.The cited prior approaches did not study specialist-training timing, and their behavior-cloning approaches can require synchronized training.Source: §2 · p. 2

Policy Distillation and Imitation Learning

  • Policy-distillation work trains a generalist from specialist teachers, using supervised demonstrations or demonstration-derived rewards; GSL uses distillation as a submodule.Source: §2 · p. 2

Large-Scale RL

  • Large-scale RL benchmarks include object or task variations and procedurally generated levels, where training one agent across variations is challenging.Source: §2 · p. 2

Multi-task RL

  • Multi-task RL often studies task encodings or gradient interference, whereas GSL also targets general environments such as Procgen whose variations lack semantic encodings.Source: §2 · pp. 2–3

3. Background

The background introduces the MDP objective and contextual MDPs as a way to represent environment variations. It also previews the paper’s motivating example and framework.

3. Background

  • An MDP specifies states, actions, transitions, rewards, and a discount factor; RL seeks a policy maximizing expected accumulated return.Source: §3 · p. 3
  • A Block Contextual MDP maps contexts to MDPs, representing variations such as objects, task semantics, or procedural-generation seeds without requiring semantic context meaning.Source: §3 · p. 3
  • The next sections first analyze a simple motivating example, then introduce ingredients of Generalist-Specialist Learning.Source: §3 · p. 3

4. An Illustrative Example

The brush-maze example illustrates how a generalist can learn shared navigation quickly yet ignore goal context, then plateau. Specialists trained on individual goals provide demonstrations that help the generalist incorporate that context and improve.

4. An Illustrative Example

  • The maze has a corridor leading to five context-specified goals; the agent observes position and velocity, controls velocity, and receives reward approximating negative geodesic distance to the goal.Source: §4 · p. 3
  • The fine-tuned generalist trained with specialist demonstrations achieves the best results efficiently in the brush-maze PPO comparison, consistently across five runs.The figure contrasts the fast-learning but quickly plateauing generalist with specialists that solve individual goals better.Source: Figure 2 · p. 3
  • The generalist quickly moves right but initially ignores goal context, often reaching the middle goal regardless of context and requiring many samples to learn context features.The authors call this “catastrophic ignorance.”Source: §4 · p. 3
  • At the plateau, five specialists trained from the generalist checkpoint master separate goal variations; their demonstrations provide rewards that help the generalist focus on context while mitigating forgetting through transferred specialized knowledge.DAPG and GAIL are given as possible ways to generate auxiliary rewards from demonstrations.Source: §4 · pp. 3–4

5. Generalist-Specialist Learning

GSL combines a generalist’s fast early learning with specialists’ mastery of smaller variation subsets, then uses specialist demonstrations to improve the generalist. This section describes the training sequence, key design choices, benchmarks, and reported results.

5.1. The Meta-Algorithm Framework

  • GSL first trains a generalist across all variations, launches specialists from its checkpoint after a suboptimal plateau, then resumes generalist training with demonstration-induced auxiliary rewards.Specialists each master a small subset of variations.Source: §5.1 · p. 4

5.2. When and How to Train Specialists

  • GSL waits for a smoothed-return plateau, then assigns the lowest-performing variations to cloned specialists; optional pre-fine-tuning helped PPO tasks in Procgen.The reported pre-fine-tuning lasts 200 epochs; specialists train in parallel with a fixed sample budget.Source: §5.1 · p. 4, §5.2 · p. 5

5.3. Generalist Training Guided by Specialist Demos

  • Specialist demonstrations, supplemented by generalist demonstrations for remaining variations, guide online DAPG or GAIL updates; offline BC was not scalable to many specialists or effective in Procgen and ManiSkill.Specialist trajectories are collected from each specialist’s best-performing checkpoint when rewards exceed a threshold.Source: §5.3 · p. 5

5.4. Benchmarks

  • Evaluation covers seven Procgen environments selected as most challenging under their setting, Meta-World MT-10/MT-50 on training environments, and a smaller ManiSkill PushChair setting with eight chairs.Procgen uses 1,024 training levels. ManiSkill evaluation is limited to PushChair because realistic simulation and point-cloud rendering are very expensive.Source: §5.4 · p. 6

5.5. Results

  • GSL outperforms PPO and PPG on Procgen training and unseen-level generalization, boosts training efficiency on MT-10/MT-50 and PushChair, and lifts plateaued generalists with specialist demonstrations.Table 1 reports training and generalization results; Table 2 covers training efficiency. The reported Procgen comparison includes 1,000 hold-out levels.Source: Table 1 · p. 7, Table 2 · p. 7, §5.5 · pp. 7–8

6. Ablation Studies

The ablations examine variation count, specialist-training timing, plateau detection, and the choice of algorithm for consolidating specialist experience. They find that the final outcome depends on when specialists start and how their demonstrations assist the generalist.

6. Ablation Studies

  • The ablations use 256 Procgen levels, rather than the 1,024 used in the main results, to obtain results faster.This is the stated setup for the ablation experiments.Source: §6 · p. 8

6.1. Influence of Environment Variations on Training Efficiency and Effectiveness

  • Training a generalist across all levels is more efficient early, while specialists trained on smaller variation sets achieve better later performance without plateauing early.The comparison uses a fixed total sample budget across configurations.Source: §6.1 · p. 8, Figure 6 · p. 8

6.2. Influence of Specialist Training Timing on Efficiency and Effectiveness

  • Training specialists after the generalist reaches a plateau is a good strategy; launching them during its early fast-learning stage worsens sample complexity.Once the BigFish generalist has trained for 2k epochs, specialist curves are nearly parallel across launch times.Source: §6.2 · p. 8, Figure 7 · p. 8

6.3. Hyperparameter Tuning of Plateau Criteria H

  • Plateau criterion H requires benchmark-dependent hyperparameter tuning, but start times 5% earlier or later yield similar final results.Tuning too early sacrifices efficiency from initial generalist learning; tuning too late prolongs the plateau and can make fine-tuning harder.Source: §6.3 · p. 9

6.5. Influence of Specialist-to-Generalist Algorithm on Final Performance

  • GSL+DAPG outperforms a single PPO generalist, while GSL+BC does not; GSL+DAPG also performs best on 1,000 novel BigFish levels.DnC RL was skipped because its quadratic computational complexity made the 75-specialist setup computationally intractable.Source: Table 3 · p. 9, §6.5 · p. 9

6.4. Diagnosis into Generalist Performance at Plateau

  • At the BigFish plateau, the generalist performs poorly on only a portion of levels, so specialists are assigned to the variants with the lowest generalist performance.The passage describes this pattern as common in Procgen environments except Plunder, where PPO struggles on a majority of levels.Source: §6.4 · p. 9

7. Conclusion

The paper presents GSL as a meta-algorithm for improving RL across many environment variations. It argues that specialists can help escape suboptimal performance plateaus, and reports improved efficiency and effectiveness on several challenging benchmarks.

7. Conclusion

  • GSL combines an RL algorithm with a learning-from-demonstrations algorithm as a meta-algorithm for RL problems with many environment variations.The paper identifies catastrophic ignorance and forgetting as challenges that may contribute to suboptimal performance plateaus.Source: §7 · p. 9
  • Introducing specialists trained on subsets of environments can help a generalist escape a suboptimal performance plateau and reach high reward.This is the paper's conclusion from its analysis of prototypical cases and experiments.Source: §7 · p. 9
  • The framework improves modern RL algorithms in efficiency and effectiveness across several popular and challenging benchmarks.Source: §7 · p. 9

A.1. Illustrative example

The illustrative example uses PPO for both generalist and specialist training, then resumes generalist training with DAPG and PPO. The appendix also identifies separate hyperparameter tables for the example and GSL.

A.1. Illustrative example

  • The illustrative example trains the generalist and specialists with PPO, then resumes generalist training using DAPG plus PPO with specialist demonstrations.Source: §A.1 · p. 12
  • Table 4 lists hyperparameters for DAPG and PPO in the illustrative example.Source: Table 4 · p. 12
  • Table 5 lists hyperparameters of GSL in the illustrative example.Source: Table 5 · p. 12

A.2. Procgen

For Procgen, the paper uses PPO or PPG for generalist and specialist training, then DAPG with the corresponding RL algorithm to resume generalist training. The appendix lists algorithm and GSL hyperparameters and notes several implementation adjustments.

A.2. Procgen

  • Procgen experiments use PPO or PPG for generalist and specialist training, then DAPG plus PPO or PPG to resume generalist training with specialist demonstrations.Source: §A.2 · p. 12
  • Tables 6 and 7 list PPO, DAPG, and PPG hyperparameters for Procgen, while Table 8 lists GSL hyperparameters.Source: Table 6 · p. 13, Table 7 · p. 13, Table 8 · p. 13
  • Procgen specialists use 16 rather than 64 parallel PPO environments, and PPG uses four policy and auxiliary epochs for sample efficiency.For Plunder, the setup trains specialists on all levels and uses more DAPG samples than in other tasks.Source: §A.2 · p. 13

A.3. Meta-World

Meta-World experiments use PPO for both generalist and specialist training, then DAPG + PPO to resume the generalist with specialist demonstrations. Specialist selection and sample use are adjusted for low-performing tasks.

A.3. Meta-World

  • Generalist and specialist policies use PPO, while specialist demonstrations guide resumed generalist training with DAPG + PPO.The policy network has two hidden layers, each with hidden size 32.Source: §A.3 · p. 13
  • Tables 9 and 10 provide the hyperparameters for PPO and DAPG in the Meta-World experiments.Source: Table 9 · p. 14, Table 10 · p. 14
  • Specialists are launched for the N lowest-performing variations, using success-rate thresholds of 0.5 for MT-10 and 0.2 for MT-50.The number of specialists varies across runs; PPO samples per epoch are reduced during specialist training to improve sample efficiency.Source: §A.3 · p. 14

A.4. ManiSkill

ManiSkill experiments use SAC for generalist and specialist training, with GAIL + SAC for resumed training from specialist demonstrations. The plateau criterion is evaluated qualitatively, with timing windows chosen to avoid slow starts and preserve time for later stages.

A.4. ManiSkill

  • ManiSkill uses the original PointNet + Transformer object-segmentation architecture, with point-cloud observations downsampled to 1200 points.Specialists focus on one chair model in PushChair.Source: §A.4 · p. 14
  • Generalist and specialist policies use SAC, and specialist demonstrations guide resumed generalist training with GAIL + SAC.Tables 11 and 12 list SAC, GAIL + SAC, and GSL hyperparameters.Source: §A.4 · p. 14, Table 11 · p. 15, Table 12 · p. 15

B. Criteria function H for performance plateaus

C. Training curves of multiple runs

Across Procgen, Meta-World, and ManiSkill, aggregated training curves show GSL consistently improving on the baseline. Curves align training stages by percentage, while smoothed endpoints may differ from batch-evaluated table results.

C. Training curves of multiple runs

  • Curves align by training percentage: generalist training occupies 0–50%, specialists 50–75%, and specialist-guided generalist training 75–100%.For Procgen and Meta-World, five runs are aggregated; ManiSkill uses three, with means and standard deviations shown.Source: C. Training curves of multiple runs · p. 15, Figure 10 · p. 16
  • Smoothed curve endpoints may not match Table 1 and 2 results, which come from batch evaluations using model checkpoints.Source: C. Training curves of multiple runs · p. 15
  • Figure 9 marks the first H(t) = 1 timestamp per run against steps, with rewards or success rate on the y-axis.It covers five Procgen and Meta-World runs and three PushChair runs.Source: Figure 9 · p. 16
  • GSL consistently improves the baseline across Procgen, Meta-World, and ManiSkill in the aggregated training curves.Figure 10 aggregates multiple runs; its y-axis uses raw episode rewards for Procgen, average success rate for Meta-World, and 1000 for ManiSkill.Source: Figure 10 · p. 16
Loading 2206.12984v1…