PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Expert Guidance for Efficient On-Policy Reinforcement Learning

Daniele Affinita, Ming Xu, Rudolf Reiter, Davide Scaramuzza, Pascal Fua

arXiv:2610.06019v1cs.LGcs.AI

TL;DR

On-policy RL requires many environment interactions, and existing approaches do not optimize an available expert’s influence using the policy’s same on-policy objective. The learner and expert alternate control in blocks, with a single expert-share parameter optimized jointly with the policy using block-level comparisons. On DMControl, the method uses a median 4.8M steps to bar versus 10.1M for PPO, a median per-task speed-up of 2.55× over PPO.

  • Problem

    On-policy RL requires many environment interactions, and existing approaches do not optimize an available expert’s influence using the policy’s same on-policy objective.

    Source: §1 · pp. 1–2

  • Method

    The learner and expert alternate control in blocks, with a single expert-share parameter optimized jointly with the policy using block-level comparisons.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    On DMControl, the method uses a median 4.8M steps to bar versus 10.1M for PPO, a median per-task speed-up of 2.55× over PPO.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    Across 34 tasks, the learner asymptotically recovers PPO performance or better, so expert suboptimality does not limit the final policy.

    Source: §1 · p. 2

Abstract

from arXiv · show

With massively parallel simulation, on-policy Reinforcement Learning methods such as PPO have become standard in many domains. However, learning from scratch is sample-inefficient and fails to exploit the potential existence of a suboptimal expert, such as a heuristic, a model-based controller, or a policy trained on a related task. Such an expert is often available and can guide early training, but its sub-optimality limits final performance. The challenge then becomes balancing expert guidance against learning from rewards. Existing methods set the expert's influence through a blending weight, a schedule, or an evaluation-driven curriculum. Alternatively, they adapt it with additional learned components such as critics over expert actions or auxiliary agents. However, none optimizes it using the same on-policy objective as the policy itself. We propose a method in which the learner and the expert alternate control within each training episode, and the expert's share of control is a single learnable parameter optimized jointly with the policy. The learner benefits from the expert early in training, but its share of control declines as the learner becomes more competent, until eventually vanishing completely. This leaves the learner acting alone and better than the suboptimal expert. We evaluate our method on 34 tasks across two benchmarks, spanning discrete and continuous action spaces, using both learned and model-based experts. Our method improves sample efficiency over guided and unguided baselines while requiring minimal hyperparameter variation. The expert's share decays to zero as the learner improves, vanishing when the expert is no longer useful.

1 INTRODUCTION

On-policy RL benefits from parallel simulation but remains sample-inefficient, while suboptimal experts can guide training without setting the learner’s final ceiling. The paper learns the expert’s share jointly with the policy and reports improved sample efficiency across 34 tasks.

Motivation

  • On-policy methods such as PPO suit massively parallel simulation, but learning from scratch requires many environment interactions and off-policy or model-based methods have higher per-interaction computation costs.The paper therefore focuses on on-policy methods for this setting.Source: §1 · p. 1
  • A fixed suboptimal expert can guide learning, but imitation alone inherits its quality; methods that retain it in deployment include residual control, action fusion, and MPC-based approaches.Examples of experts include approximate-model controllers, hand-engineered policies, previously trained agents, and agents from related tasks.Source: §1 · p. 1

Research gap

  • Existing approaches schedule, evaluate, or otherwise adapt expert influence, but the paper identifies no prior method that optimizes it with the policy’s same on-policy objective.Some adaptive approaches use a learned selector, a critic over expert actions, or a reward-only counterfactual agent.Source: §1 · p. 2

Contribution

  • The proposed method jointly optimizes a single expert-share parameter with the policy; block-level comparisons, an expert-use price, and imitation on expert blocks support eventual withdrawal.As the learner catches up, its expert share falls to zero and training becomes standard policy gradient.Source: §1 · p. 2

Results

  • Across 34 tasks, the method improves sample efficiency over guided and unguided baselines, while the learner asymptotically matches or exceeds PPO and weaker experts are withdrawn faster.The evaluation spans two benchmarks and discrete and continuous action spaces, with minimal hyperparameter variation.Source: §1 · p. 2

2 RELATED WORK

Prior work uses experts as action targets, objective terms, or policies that control part of a rollout. This method belongs to the rollout-guidance family and learns the expert’s share from block-level comparisons.

Overview

  • The paper groups prior work by whether the expert supplies target actions, contributes an objective term, or controls part of the rollout.Its learned expert share addresses how strongly and how long rollout guidance influences the learner.Source: §2 · p. 2

Imitating an expert

  • Imitation methods can inherit a fixed expert’s ceiling, while guided policy search and expert iteration use teachers that improve; LOKI, MAMBA, and RPI combine imitation with reward-driven updates.RPI is closest in setting, but compares value functions learned from each oracle’s rollouts rather than using the learner’s critic on training rollouts.Source: §2 · p. 2

Expert-regularised reinforcement learning

  • Expert-regularized methods add expert-derived losses or terms to the objective, and their design question is the regularizer’s weight.The Q-filter applies imitation where a critic prefers the demonstrated action, while TGRL compares guided and reward-only agents to adapt the weight.Source: §2 · p. 2

Experts that act during training

  • Rollout-guidance methods shrink expert control through a curriculum or schedule, whereas this method uses fixed-length blocks and a state-independent switch.Its share gradient compares expert and learner block advantages from training rollouts using the learner’s critic, without a selector, expert-action critic, evaluation phase, or auxiliary agent.Source: §2 · pp. 2–3, Figure 1 · p. 3

Temporal abstraction

  • The block structure can be viewed through the options framework: expert and learner are fixed-duration options, but only the learner is trained.The learner update corresponds to the intra-option policy gradient, while this method learns to withdraw one option.Source: §2 · p. 3

3 METHOD

The method alternates a fixed expert and learner in blocks and jointly learns the policy and expert share. Block-level return comparisons, a cost for expert use, and imitation on expert blocks guide the handover to learner-only training.

3 METHOD

  • The expert and learner alternate control in fixed-length blocks, and the expert share is automatically adjusted as the on-policy learner improves.Expert blocks can bring the agent to states it may not reach alone, including reward-bearing states in sparse-reward tasks.Source: §3 · p. 3, Figure 2 · p. 4

3.1 PROBLEM FORMULATION

  • At each block boundary, an independent Bernoulli switch with probability λ selects the fixed expert or learner for the entire k-step block; λ is constrained below 1 so the learner can act.The mixed rollout objective equals the learner’s objective when λ = 0.Source: §3.1 · p. 4

3.2 POLICY GRADIENT ESTIMATOR UNDER BLOCK ROLLOUTS

  • Because switches are state-independent, expert and learner blocks are compared from the same boundary-state distribution; the λ-gradient raises expert share when expert-block advantage exceeds learner-block advantage.Blocks make that comparison more informative than switching at every step, where outcomes depend on subsequent actions.Source: §3.2 · p. 5

3.3 PENALTY ON EXPERT USE AND IMITATION

  • A price on expert use makes the share decrease once the expert’s advantage falls below a margin, while imitation on expert blocks also trains the learner from those states.The imitation term acts only on expert blocks and vanishes at λ = 0, when training is standard policy gradient.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

Across two benchmarks, the experiments compare learned expert handover with guided and unguided baselines, test whether handover responds to expert quality, and examine key hyperparameters. The method improves sample efficiency while adapting expert use to task and expert quality.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers 34 tasks across DMControl and MiniGrid, with continuous and discrete actions, five seeds per task, and 20M environment steps per method.Experts include noisy DFS planners, early PPO checkpoints, TD-MPC2 policies, and inaccurate-model controllers.Source: §4.1 · pp. 6–7

4.2 COMPARISON OF RETURN PERFORMANCE

  • On DMControl, our method has the lowest median steps to bar (4.8M) versus PPO (10.1M), highest mean normalized return, and is fastest on 16 of 25 tasks.It beats every baseline on both metrics in every task family except Finger and steps to bar in MiniGrid.Source: §4.2 · p. 7, Table 1 · p. 8
  • On MiniGrid, our method has the highest final return on 4 tasks and ties for highest on 3 more, avoiding BC→PPO’s collapse on the largest KeyCorridor variants and DoorKey-Random-16x16.PPO reaches the bar on only 3 of 9 tasks; no single linear schedule horizon works across all tasks.Source: §4.2 · p. 7
  • With no hyperparameter changes, λ reaches zero at 9.8M steps for an exact MPC model and 2.2M for b = −0.9; the learner ends above PPO even at that mismatch.Intermediate mismatch levels have intermediate withdrawal times; learner return is about 0.7 normalized except at b = −0.9.Source: §4.2 · p. 8, Figure 4 · p. 8

4.4 CAN A FAILING EXPERT HELP?

  • On Frozen Lake with pslip = 0.3, our learner reaches the goal in 98.5% of episodes, versus 4% for Kickstarting, 28% for JSRL, and 0% for BC→PPO.The learner retains the expert’s route while learning to avoid holes; PPO never leaves the start.Source: Figure 5 · p. 8, §4.4 · p. 9

4.5 ABLATIONS

  • Across Humanoid tasks, removing imitation leaves the learner at the floor, while cI = 0.1 gives the best trade-off because excessive imitation anchors it to the expert and hinders exploration.The sweep tests cI ∈ {0, 0.01, 0.1, 1}.Source: §4.5 · p. 9, Figure 6 · p. 9
  • At k = 1, the block advantage gap signal-to-noise is near zero or negative; d(k)/√k peaks at intermediate lengths, k ≈75–300, with k = 100 near the peak.Longer blocks improve per-block visibility, but fewer blocks fit in a fixed batch.Source: §4.5 · p. 9
  • At cE = 0, λ remains positive on 13 of 25 DMC tasks even after the learner matches the expert; cE = 0.3 peaks in relative return, while higher prices can withdraw the expert too early.Mean withdrawal falls from 17.4M steps at cE = 0 to 0.6M at cE = 3; cE = 0.3 is used on every task without per-task tuning.Source: §4.5 · p. 9, Figure 6 · p. 9

5 CONCLUSION

The method shares each episode in blocks between a fixed expert and the learner, optimizing the expert’s share with the policy’s objective. It reaches a given return in fewer environment steps than guided and unguided baselines, and the expert can be removed once its share reaches zero.

5 CONCLUSION

  • A median per-task speed-up of 2.55× over PPO on DMControl accompanies fewer environment steps to a given return than guided and unguided baselines.The largest margins occur under sparse rewards, even with an expert that never succeeds.Source: §5 · p. 9
  • Committing to blocks reduces the λ-gradient to the gap between the returns earned by expert and learner blocks.A price on expert use makes withdrawal part of the objective, while imitation on expert blocks lets every block train the learner.Source: §5 · p. 9
  • Once λ reaches zero, the deployed policy runs without the expert; the handover needs no per-task tuning.The authors propose extending block-advantage comparisons to multiple experts and using λ to budget queries to expensive experts.Source: §5 · p. 9

A PROOF OF PROPOSITION 1

The proof evaluates the expectation in equation 5 by conditioning in stages: first on the boundary state, then on the independent switch, and finally on block execution. It uses the switch-dependent score structure to simplify the expectation before substituting across blocks.

A PROOF OF PROPOSITION 1

  • The expectation in equation 5 is taken first over the boundary state under the mixture, then over the independent Bernoulli switch, and then over block execution.The switch remains unchanged by conditioning because it is independent of the boundary state.Source: A PROOF OF PROPOSITION 1 · p. 13
  • Because the score depends only on the switch, it can be moved outside the innermost expectation.Source: A PROOF OF PROPOSITION 1 · p. 13
  • Averaging over the switch cancels each outcome’s probability against its own branch denominator, and substitution for every block gives equation 10.Source: A PROOF OF PROPOSITION 1 · p. 13

B ADDITIONAL RESULTS

The additional-results appendix summarizes per-task results by benchmark and task family, alongside task-level DMControl tables and learning-curve and expert-share figures. Aggregate metrics include steps to a performance threshold and normalized return.

B ADDITIONAL RESULTS

  • Across all 34 tasks, the appendix groups DMControl into Low-dim. control, Finger, and Locomotion families and reports MiniGrid and all-task aggregates.The table’s aggregate measures are median steps to bar and mean normalized return over tasks; runs not reaching the bar within 20M steps count as 20M.Source: B ADDITIONAL RESULTS · p. 13

B.1 DMCONTROL

  • For each DMControl task, Table 2 reports steps to bar and asymptotic return, while Figures 7 and 8 show learner learning curves and the method’s expert share λ.Source: §B.1 · p. 13

B.2 MINIGRID

The MiniGrid results section points to task-level performance and learning-curve plots, as well as plots of the method’s expert share over training. The supplied passages identify these displays but do not state their outcomes.

B.2 MINIGRID

  • Table 3 reports steps to bar and final return for every MiniGrid task.Steps to bar measures steps to reach 75% of the task’s best final return; final return is measured deterministically after 20M steps.Source: §B.2 · p. 13, Table 3 · p. 17
  • Figure 9 plots per-task learner returns for each method, with the expert’s return marked by a dotted line.Source: Figure 9 · p. 16
  • Figure 10 plots the method’s expert share λ over training for each MiniGrid task.Source: Figure 10 · p. 16

B.3 FROZENLAKE

FrozenLake tests guidance in a slippery gridworld with a long, hole-filled route. At pslip = 0.3, the proposed learner reaches the goal reliably while the listed baselines do not.

B.3 FROZENLAKE

  • The 36 × 36 grid has 32.9% holes, and five walls force a 123-cell serpentine route with 25 turns.The two-dimensional acceleration can be rotated by a quarter turn with probability pslip; entering a hole ends the episode.Source: §B.3 · p. 18
  • Methods share the environment, expert, network, PPO hyperparameters, and 2 × 10^7-transition budget; success is measured with the learner acting alone using deterministic actions.Source: §B.3 · p. 18
  • At pslip = 0.3, the learner reaches the goal in 98.5% of episodes, while PPO never reaches it and the other methods usually fall into holes.Table 4 reports final success rate on FrozenLake at pslip = 0.3, averaged over 5 seeds.Source: §B.3 · p. 18, Table 4 · p. 18

B.4 ABLATIONS PER TASK

Per-task ablations show how imitation weight and expert price affect returns and expert withdrawal. The hyperparameter tables describe shared backbones and guide settings.

B.4 ABLATIONS PER TASK

  • With cI ∈ {0, 0.01}, return stays near zero on the three Humanoid tasks; cI = 1 tracks the expert on FingerSpin but ends lower on Humanoid, with little effect on BallInCup and WalkerWalk.Source: Figure 11 · p. 19
  • With cE = 3, expert share λ reaches zero within 1M steps on all three tasks; with cE = 0, it remains positive through training on WalkerWalk and CheetahRun.On FingerSpin with cE = 0, λ reaches zero after about 10M steps.Source: Figure 12 · p. 19

C HYPERPARAMETERS

D GUIDES

The guide-quality tables report guide rewards for DMControl and MiniGrid and characterize the guides as suboptimal relative to task performance ceilings.

D GUIDES

Loading 2610.06019v1…