PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Fidelity Policy Gradients Stabilize Data-Scarce Reinforcement Learning

Xinjie Liu, Ruihan Zhao, Anirban Chaudhuri, Cyrus Neary, Ufuk Topcu, David Fridovich-Keil

arXiv:2610.02505v1cs.LGcs.AIcs.RO

TL;DR

Scarce, expensive target-domain data can make on-policy policy-gradient learning unstable, while existing multi-fidelity methods often optimize biased LF objectives and prior MFPG work was limited to REINFORCE. This paper develops MFPG-PPO and a budget-aware variant that use LF data as control variates with correlation-preserving sampling and uncertainty checks. Across simulated tasks it improves over HF-only PPO in nearly all settings, and it supports Franka-arm learning with 4 episodes per update.

  • Problem

    Scarce, expensive target-domain data can make on-policy policy-gradient estimates too noisy for stable learning, and prior MFPG work was limited to REINFORCE and small-scale simulations.

    Source: §1 · p. 1

  • Method

    MFPG-PPO uses abundant LF data to construct a control variate for HF policy-gradient estimates, redesigning sampling and advantage estimation and monitoring uncertainty; a budget-aware variant allocates samples across fidelities.

    Source: §1 · p. 2

  • Results

    MFPG-PPO improves over HF-only PPO in nearly all tested settings and enables stable Franka-arm learning with 4 episodes per policy update and no human demonstrations.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    MFPG-PPO uses LF data to reduce variance in HF estimates rather than directly optimizing biased LF objectives, supporting on-policy learning with very small target-domain batches.

    Source: §1 · p. 1, §6 · p. 10

    Re-synchronizing coupled LF rollouts while uncoupled rollouts evolve freely can introduce bias when their LF statistics differ in mean; achieving full variance reduction without bias remains open.

    Source: §6 · p. 10

Abstract

from arXiv · show

Policy gradient methods for on-policy reinforcement learning (RL) can become unstable when expensive, scarce target-domain data yield noisy gradient estimates. We address this challenge by complementing limited high-fidelity (HF) target-domain data with abundant, cheap, but biased low-fidelity (LF) data, e.g., from a simplified simulator. Most existing methods directly optimize biased objectives based on LF data. In contrast, the recently introduced multi-fidelity policy gradient (MFPG) framework uses LF data solely to construct a control variate that reduces variance and improves HF data efficiency without biasing the policy gradient estimator. However, published work on MFPG is limited to REINFORCE on small-scale simulation tasks. We develop MFPG for modern actor-critic learning in GPU-parallel simulation and on a physical robot. Our analysis and experiments show that naive extensions to proximal policy optimization (PPO) can lose cross-fidelity correlation or inflate variance. Our MFPG-PPO addresses these failures by redesigning the sampling, advantage estimation, and control variate construction to preserve cross-fidelity correlation, and by monitoring estimator uncertainty to prevent variance inflation. We also introduce a budget-aware MFPG-PPO to divide a fixed sampling budget among high- and low-fidelity data sources. Across simulated robot locomotion tasks of varying LF-to-HF transfer difficulty and HF data budgets, MFPG-PPO improves upon PPO trained on HF data alone in nearly all settings, and consistently matches the performance of PPO trained with 16x more HF data on the hardest task at the smallest HF budgets. In contrast, most baselines that use LF data perform well only where direct LF-to-HF transfer succeeds. MFPG-PPO enables stable learning on a physical Franka arm using only 4 real-robot episodes per update and no human demonstrations.

1 INTRODUCTION

On-policy policy gradients can become unstable when scarce, expensive target-domain data make gradient estimates too noisy. MFPG-PPO uses abundant, biased low-fidelity data to reduce variance while retaining an HF-targeted update, and improves on HF-only PPO across nearly all tested settings.

1 INTRODUCTION

  • Earlier MFPG work was limited to REINFORCE and small-scale single-environment simulations, while naive PPO substitutions could underperform single-fidelity PPO.LF-only objective optimization can also bias updates and degrade HF performance.Source: §1 · p. 2
  • MFPG-PPO extends multi-fidelity policy gradients to PPO by preserving cross-fidelity correlation and monitoring estimator uncertainty.The budget-aware variant allocates a fixed sampling budget across HF and LF data sources.Source: §1 · p. 2
  • MFPG-PPO improves over HF-only PPO in nearly all simulation settings, is never distinguishably worse, and scales monotonically with more HF data.Most LF-data baselines perform well only where direct LF-to-HF transfer succeeds.Source: §1 · pp. 2–3
  • On the hardest task at the smallest HF budgets, MFPG-PPO consistently matches the mean return of PPO trained with 16× more HF data.Figure 1 reports the rough-terrain humanoid comparison at matched HF budgets.Source: Figure 1 · p. 2, §1 · p. 3
  • MFPG-PPO enables stable learning on a physical Franka arm with only 4 episodes per policy update and no human demonstrations.The work also targets data batches up to 512× smaller than those in common GPU-simulation RL pipelines.Source: §1 · pp. 1, 3

2 RELATED WORK

Prior multi-fidelity RL work combines data through pooling, correction, or control variates, while another line studies which fidelity to query at each training stage. This paper builds on multi-fidelity control variates for actor-critic learning.

2 RELATED WORK

  • Off-dynamics RL corrects estimated dynamics mismatch through reward penalties, representation mismatch, filtering, support extension, or imitation-based transfer.The cited work also extends to offline and hybrid offline-online settings.Source: §2 · p. 3
  • Pooling and off-dynamics correction optimize biased LF-based objectives, whereas multi-fidelity control variates use LF data to reduce variance in HF estimates.Prior control-variate work addressed tabular value estimation or REINFORCE; this paper extends the approach to actor-critic learning in GPU-parallel simulation and on a real robot.Source: §2 · p. 3

3 PRELIMINARIES

The paper models high- and low-fidelity environments as discounted MDPs that share state and action spaces, initial-state distributions, and discount factors but may differ in dynamics and rewards. MFPG uses paired HF and LF statistics to form an unbiased, potentially lower-variance HF gradient estimate.

3 PRELIMINARIES

  • HF and LF MDPs share state and action spaces, initial-state distribution, and discount factor, but may differ in transition distributions and reward functions.The optimization target is expected discounted return in the HF environment.Source: §3 · p. 3
  • PPO estimates policy gradients from transition-level advantages, commonly computed with GAE over short rollout segments and value-function bootstrapping.REINFORCE instead uses trajectory-level Monte Carlo returns and requires complete episodes.Source: §3 · p. 3
  • MFPG adds a centered LF statistic to the HF statistic as a control variate, preserving the HF statistic’s mean while reducing variance when the paired statistics are correlated.The optimal coefficient depends on their covariance and the LF statistic’s variance.Source: §3 · p. 4

4 APPROACH

MFPG-PPO adapts control variates to PPO by pairing HF and LF statistics, preserving their correlation, and applying the correction only when evidence supports it. Its budget-aware variant allocates sampling across fidelities under a fixed per-iteration cost.

4 APPROACH

  • Naive PPO extensions can lose cross-fidelity correlation or inflate variance, so MFPG-PPO redesigns sampling, advantage estimation, and control-variate construction while monitoring uncertainty.When checks fail, it sets the coefficient to zero and falls back to HF-only PPO.Source: §4 · p. 4, §4.1 · p. 4

4.1 MFPG-PPO

  • HF and coupled LF rollouts share controllable randomness and periodically re-synchronize, limiting their divergence and preserving paired-statistic correlation.Uncoupled LF rollouts provide additional LF data; Figure 2 depicts the three batches.Source: §4.1 · p. 5, Figure 2 · p. 5
  • Coupled LF GAE horizons match HF horizons; on a robot, LF segments are stitched to match each complete HF episode’s horizon.Separate LF critics accommodate the different state distributions of coupled and uncoupled LF rollouts.Source: §4.1 · p. 5
  • Preserving correlation can introduce bias because coupled and uncoupled LF statistics come from different trajectory distributions.Re-synchronizing both LF batches would remove this bias but leave some variance irreducible.Source: §4.1 · p. 5
  • Per-step coefficient fitting increases sample size but can omit cross-step covariance; episode-sum fitting retains it, so the method uses per-step fitting in simulation and episode sums on the robot.On the robot task, the reported correlation is about 0.17 per step versus about 0.6 for episode sums.Source: §4.1 · p. 6
  • The control variate is applied only when paired-statistic correlation is distinguishable from zero and the corrected statistic has lower sample variance.The correlation check uses a 90% confidence level; otherwise the update falls back to HF-only PPO.Source: §4.1 · pp. 4, 6

4.2 BUDGET-AWARE MFPG-PPO

  • Budget-aware MFPG-PPO chooses HF and LF sample counts to minimize estimator variance under a fixed budget with heterogeneous per-sample costs.Its idealized optimal LF-to-HF sample ratio grows as LF rollouts become cheaper or more strongly correlated with HF rollouts.Source: §4.2 · p. 6
  • The method updates its allocation as the policy changes; when the optimal LF-to-HF ratio is one, it collects coupled pairs and falls back to HF-only PPO.Coefficient fitting and monitoring otherwise carry over unchanged.Source: §4.2 · pp. 6–7

5 EXPERIMENTS

The experiments evaluate MFPG-PPO across simulated locomotion tasks, HF data budgets, and real-robot cube picking. Results compare its reliability and scaling with HF-only and LF-using baselines, then examine real-robot learning and variance-control ablations.

5 EXPERIMENTS

  • The experiments judge methods by monotone improvement as HF data increase and reliability: better than HF-only when LF data help, and no worse when they mislead.MFPG-PPO-FR’s mean return rises with more HF data in every task, plateauing only at the largest H1 Rough budget.Source: §5 · p. 7, §5.3 · p. 8

5.1 EXPERIMENTAL SETUP

  • The evaluation spans H1 and ANYmal-D locomotion, HF budgets from 8 to 512 parallel environments, and Franka cube picking, with comparisons against PPO HF-Only and LF-data baselines.Locomotion policies are evaluated over 2048 HF episodes; the robot study uses three seeds with 80 iterations of four episodes.Source: §5.1 · p. 7

5.2 HOW DOES MFPG-PPO SCALE WITH HF DATA SCARCITY?

  • At the three smallest HF budgets on H1 Rough, MFPG-PPO-FR matches or exceeds PPO HF-Only trained with 16× more HF data.PPO HF-Only collapses to near-zero return at budgets of 1/128× and below, while MFPG-PPO-FR reaches a mean return around 40 at 1/256×.Source: §5.2 · p. 7

5.3 HOW DO DIFFERENT USES OF LF DATA SCALE ACROSS TASKS AND HF BUDGETS?

  • MFPG-PPO-FR outperforms PPO HF-Only in 14 of 16 task-budget settings, while MFPG-PPO-BA outperforms it in all 16.Figure 3 reports differences in mean HF evaluation return relative to PPO HF-Only; greedier methods gain more when LF-to-HF transfer is easy.Source: Figure 3 · p. 8, §5.3 · p. 8

5.4 CAN MFPG-PPO LEARN STABLY ON A REAL ROBOT?

  • On Franka cube picking, MFPG-PPO-FR raises final real-robot success from 40.0% to 91.7%, versus 55.0% for PPO HF-Only.Every MFPG-PPO-FR seed reaches 90–95%, exceeding every PPO HF-Only seed at 30–70%.Source: §5.4 · p. 8, Figure 4 · p. 9

5.5 ABLATIONS: NAIVE CONTROL VARIATE CONSTRUCTION AMPLIFIES VARIANCE

  • Per-minibatch coefficient re-fitting and monitoring act jointly: removing both causes learning to collapse at the 1/8× HF budget.Without monitoring, variance ratios exceed 1 in most iterations and sometimes exceed 1000; the full method exceeds 1 in only a small fraction.Source: §5.5 · p. 9, Figure 5 · p. 9

6 CONCLUSIONS AND DISCUSSION

MFPG-PPO and its budget-aware variant stabilize data-scarce on-policy RL, including with four real-robot episodes per update. The conclusion also identifies open design challenges, while the paper reports its AI use and reproducibility materials.

6 CONCLUSIONS AND DISCUSSION

  • MFPG-PPO improves on HF-only PPO in nearly all tested settings, is never distinguishably worse, and scales monotonically with more HF data.It supports stable learning with as few as 4 real-robot episodes per update or target-domain batches up to 512× smaller than common GPU-simulation RL pipelines.Source: §6 · p. 10
  • Avoiding bias while keeping both variance components reducible remains an open challenge; increasing cross-fidelity correlation and constructing coupled LF simulators are future directions.Re-synchronizing coupled LF rollouts to HF states while uncoupled rollouts evolve freely can introduce bias if the two LF statistics have different means.Source: §6 · p. 10

AI USE STATEMENT

  • The authors used generative AI for implementation, data preparation, results interpretation, proof assistance, methodology and experiment feedback, literature work, writing, and figure or image work.They report not using AI to generate synthetic datasets, develop theoretical models, formulate mathematical claims, refine hypotheses, or identify research topics or gaps.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The authors report reviewing and testing AI-written code, checking AI-assisted proof steps, and verifying experimental results against logs; they will release the software upon publication.Appendix materials cover algorithm details, monitoring checks, hyperparameters, implementation, task protocols, proofs, and budget allocation.Source: AI USE STATEMENT · p. 10, REPRODUCIBILITY STATEMENT · p. 10

A.1 PROOF OF LEMMA 1

The proof derives the variance-minimizing control-variate coefficient and compares it with a pooled per-step fit. Its result depends on stated regularity and constant-mean conditions.

A.1 PROOF OF LEMMA 1

  • The proof assumes finite second moments, positive variance of Blow, and coupling at every episode step so the pair set contains exactly KH pairs.Source: §A.1 · p. 15
  • The variance of the estimator is a quadratic in c, uniquely minimized at c∗ = −Cov(Bhigh, Blow)/Var(Blow).The proof uses independence across episodes and treats the LF mean as contributing no variance.Source: §A.1 · p. 15
  • With constant per-step means, the pooled per-step fit converges almost surely to cstep as the number of episodes grows.The fit pools KH pairs and is the ratio of the estimated per-step covariance to the estimated per-step variance, with a negative sign.Source: §A.1 · p. 15
  • The global and per-step coefficients need not match: for H = 2, same-step covariances can vanish while Blow = Bhigh yields cstep = 0 and c∗ = −1.Source: §A.1 · p. 16
  • Constant per-step means isolate dependence structure; without that assumption, the limiting fit also reflects variation in means over the horizon.Source: §A.1 · p. 16

A.2 WHY ARE THE UNCOUPLED LF ROLLOUTS NOT RE-SYNCHRONIZED?

Free-running uncoupled LF rollouts let the control variate address both between-state and within-state variance, unlike a re-synchronized LF baseline. This choice can introduce bias when the coupled and uncoupled LF statistics have different means.

A.2 WHY ARE THE UNCOUPLED LF ROLLOUTS NOT RE-SYNCHRONIZED?

  • The free LF baseline is the mean of uncoupled LF statistics, while the re-synchronized baseline is their conditional mean given C; both baselines are treated as exact.Source: §A.2 · p. 16
  • A re-synchronized LF baseline cannot reduce between-state variance, whereas a free LF baseline can let one coefficient reduce both between-state and within-state variance.The between-state component reflects which states the scarce HF rollouts start from; the re-synchronized correction acts only on within-state variance.Source: §A.2 · p. 16
  • In the ideal case Xlow = Xhigh with the uncoupled LF mean equal to E[Xhigh], c = −1 makes Zu have zero variance, while ZC retains the full between-state variance.Source: §A.2 · p. 16
  • Leaving uncoupled rollouts free keeps both variance components reducible, but may bias the estimator unless coupled and uncoupled LF statistics have the same mean.Removing that bias by re-synchronizing both LF batches would leave variance associated with re-synchronization states irreducible; eliminating both variance components without bias would require additional HF states.Source: §A.2 · p. 16

A.3 BUDGET ALLOCATION

The allocation analysis chooses high- and low-fidelity sample counts under a fixed budget to minimize the control-variate estimator’s variance. Its optimality result is restricted to an idealized scalar estimator and paired-sample layouts.

A.3 BUDGET ALLOCATION

  • The coefficient is minimized for fixed counts before substitution into the allocation objective; the sampling model represents each sample as a T-step rollout with a mean surrogate.The paired HF and LF surrogates have correlation ρ, while additional LF samples share the paired LF marginal distribution.Source: §A.3 · p. 17
  • The allocation fixes ν = mlow/mhigh ≥ 1 and expresses mhigh through the budget and per-sample costs before minimizing estimator variance.The counts are HF samples and total coupled plus uncoupled LF samples.Source: §A.3 · p. 17
  • The continuous optimum chooses the low-to-high sample ratio subject to ν ≥ 1; at ν = 1, the LF correction vanishes, and a ratio above 1 alone does not ensure lower variance than equal-cost HF-only sampling.For ν > 1, the solution agrees with Peherstorfer et al. (2016, Thm. 3.4), while the equal-cost comparison is qualified by Peherstorfer et al. (2016, Corollary 3.5).Source: §A.3 · p. 17
  • Flooring continuous counts preserves the budget cap and ordering but can leave budget unused or yield zero HF samples, so counts are repaired and infeasible budgets rejected.Unused budget is not redistributed; the repaired counts map to N high = mhigh and N low = mlow − mhigh.Source: §A.3 · p. 17
  • The variance optimum applies only to the idealized scalar estimator: deployed coupled and uncoupled LF rollouts may have different marginals, and per-transition correlations omit cross-step covariance.The moments are also estimated and smoothed online as the policy changes.Source: §A.3 · p. 17

A.4 COMPLETE ALGORITHM

MFPG-PPO collects HF, coupled LF, and uncoupled LF data, constructs minibatch statistics and a control-variate objective, then updates the policy with PPO. Its budget-aware variant allocates rollout counts before collection.

A.4 COMPLETE ALGORITHM

  • Each minibatch uses Eq. (1) with its HF term averaged only over transitions that have valid coupled LF counterparts.The resulting objective is used for a PPO step, while each critic is fit on its own transitions with targets that do not span a re-synchronized state.Source: §A.4 · p. 18
  • Each iteration collects HF rollouts, paired LF rollouts reset to matching HF starts and driven by recorded noise, plus uncoupled LF rollouts with free resets.The algorithm forms paired and unpaired transition sets for subsequent estimation.Source: §A.4 · p. 18
  • The algorithm estimates advantages and surrogate statistics for the batches, then fits and smooths the control-variate coefficient on paired data and monitors whether to apply it.At minibatch level it refits the coefficient and applies the correction only if all checks pass; otherwise it sets c = 0.Source: §A.4 · p. 18
  • The budget-aware variant sets HF and LF rollout counts before each iteration’s collection and replaces the LF baseline mean with the mean over all LF transitions in each minibatch.Its counts follow the allocation and integer-rounding procedure described in Appendix A.3.Source: §A.4 · p. 18

A.5 MONITORING THE CONTROL VARIATE

The monitoring procedure applies the control variate only when estimated evidence supports it and its variance check passes. Simulation uses batch and minibatch checks, while the robot accumulates correlation evidence across iterations.

A.5 MONITORING THE CONTROL VARIATE

  • Simulation disables the control variate for an iteration if the batch-level variance-reduction check fails, and uses a paired-sample bootstrap interval to gate each eligible minibatch.If the minibatch interval contains zero, the correction is disabled; otherwise its smoothed coefficient is applied only when the variance check passes.Source: §A.5 · p. 18
  • With only 4 paired robot episodes per iteration, MFPG-PPO accumulates correlation evidence across recent iterations using Fisher-z intervals and disables the correction when an interval contains zero.The coefficient is updated at iteration start and, when the interval excludes zero, before minibatch updates.Source: §A.5 · p. 19
  • The variance check compares corrected and HF-only sample variances, setting c = 0 when their ratio is at least 1 and thereby reverting to an HF-only actor objective.Simulation checks individual paired steps; the robot check uses sums over paired episodes.Source: §A.5 · p. 19
  • Table 1 reports the control-variate monitoring hyperparameters.Source: Table 1 · p. 19

A.6 IMPLEMENTATION AND EXPERIMENT DETAILS

This appendix section documents implementation, tuning, and experimental protocols for the simulation and Franka studies, alongside additional results and ablation diagnostics. It also describes cross-fidelity correlation and variance patterns and how diagnostic curves and uncertainty bands are displayed.

A.6.1 HYPERPARAMETER TUNING PROTOCOL

  • Hyperparameters are tuned on H1 Rough and ANYmal-D Rough with N_high = 256 and 3 seeds, then fixed across locomotion tasks and HF budgets.The protocol selects configurations using standardized final-return and AUC metrics; the PPO backbones and method-specific parameters are searched in separate stages.Source: §A.6.1 · p. 19

A.6.2 IMPLEMENTATION DETAILS

  • MFPG-PPO uses an HF worker and LF learner: returned HF states initialize coupled LF rollouts before the learner collects LF data and updates the policy and critics.Coupled environments match locomotion commands and terrain assignments, or the measured object pose for manipulation.Source: §A.6.2 · p. 20

A.6.3 EXPERIMENT DETAILS

  • HF-data on-policy methods share LF-pretrained policy initialization, and simulation returns are evaluated deterministically over 2048 completed episodes with means and 95% bootstrapped confidence intervals.Training runs for 8000 iterations on H1 Rough and 4000 on other locomotion tasks, excluding pretraining.Source: §A.6.3 · pp. 24–25
  • The Franka task uses 4 physical episodes per update, with MFPG-PPO also collecting 20-step LF rollouts from 8192 parallel environments; each method trains for 80 iterations across 3 seeds.About 69% of training control steps lack a valid cube pose estimate, and friction and controller stiffness also differ across fidelities.Source: §A.6.3 · pp. 25–26

A.7 EXTENDED EXPERIMENTAL RESULTS

  • MFPG-PPO achieved higher mean return than PPO HF-Only at all 5 smallest HF budgets after extensive LF pretraining, while PPO HF-Only collapsed to near-zero mean return at the 2 smallest budgets.Figure 8 also reports absolute mean HF evaluation returns for the settings compared by return differences in Figure 3.Source: §A.7 · p. 26, Figure 9 · p. 27, Figure 8 · p. 27
  • Per-step paired-statistic correlations are generally low on Franka, but episode aggregation captures cross-step covariance; locomotion event timing aligns more closely across fidelities.Figure 11 reports event-timing differences and paired-statistic correlations across tasks.Source: §A.7 · p. 26, Figure 11 · p. 28
  • Without monitoring, MFPG-PPO variants amplify variance in most iterations at the 1/32 HF budget on H1 Rough.The diagnostic tracks the fraction of iterations where the variance ratio exceeds 1, 10, and 1000.Source: §A.7 · p. 29, Figure 13 · p. 29
  • The ablation variants exhibit the same level of cross-fidelity correlation.Source: §A.7 · p. 29
  • Minibatch-rejection and correlation curves are smoothed with a centered 800-iteration median; bands show 95% bootstrapped confidence intervals for seed means.Source: §A.7 · pp. 29–30
  • Figure 14 tracks variance ratio, monitoring-check failure fraction, and estimated paired-statistic correlation for ablations at 1/8× and 1/32× HF budgets on H1 Rough.Source: §A.7 · pp. 29–30, Figure 14 · p. 30
Loading 2610.02505v1…