PaperlayerView exact revision on arXiv

Source-linked AI summary

Pessimistic Latent Task-aware Optimization for Robust Offline Meta-Reinforcement Learning

Chang-Hoon Jeong, Kisung Shin, Hanwool Sul, Byoung-Tak Zhang

arXiv:2610.15428v2cs.LGcs.AI

TL;DR

In COMRL, improving task representations does not expose the policy during training to off-support latents produced by parameter-extrapolation tasks. PLATO perturbs inferred latents outward and trains the policy on pessimism-weighted imagined rollouts; across eight MuJoCo benchmarks, it improves out-of-support returns on every environment without sacrificing in-distribution performance.

  • Problem

    Representation-focused COMRL leaves the policy untrained on latents outside the training support, which parameter-extrapolation tasks produce.

    Source: §1 · p. 2

  • Method

    PLATO uses an outward-from-centroid manifold assumption to synthesize off-support latents and trains the policy on imagined rollouts weighted by ensemble disagreement.

    Source: §1 · p. 2, §3.5 · p. 6

  • Results

    PLATO improves out-of-support returns on every environment across eight MuJoCo benchmarks without sacrificing in-distribution performance.

    Source: §7 · p. 10

  • Takeaways & Limitations

    The reported results show that training-time exposure to synthesized off-support latents can accompany stronger out-of-support returns while preserving in-distribution performance.

    Source: §7 · p. 10

    PLATO’s geometric assumption is that task latents lie on a low-dimensional manifold and out-of-support latents lie farther from the training centroid on average.

    Source: §1 · p. 2, §3.2 · p. 4

Abstract

from arXiv · show

Context-based offline meta-reinforcement learning (COMRL) has advanced almost entirely through better task representations, which separate task identity from artifacts of the behavior policy so that the latent reflects the task alone. The implicit expectation has been that a clean representation will transfer to out-of-support (parameter-extrapolation) tasks. On such a task, however, the encoder produces a latent outside the training support, where the policy has never been trained. We address this gap with Pessimistic Latent Task-aware Optimization (PLATO), which exposes the policy to off-support latents during training. PLATO assumes a mild manifold hypothesis: task latents lie on a low-dimensional manifold, and out-of-support tasks lie farther from the training centroid. It exploits this geometry by perturbing inferred latents outward from the centroid to synthesize counterfactual tasks. A learned decoder ensemble then rolls out short trajectories at each perturbed latent, and the policy is updated on the imagined transitions under a pessimism weight that down-weights steps on which the ensemble disagrees. We prove that the perturbation leaves the training support by a controllable margin and that the out-of-support generalization gap is bounded by the same perturbation reach, and we give explicit conditions, verified empirically, under which the epistemic disagreement at the perturbed latent stays bounded away from zero. On eight MuJoCo continuous-control benchmarks, PLATO consistently improves out-of-support returns over strong COMRL baselines while remaining competitive on in-distribution tasks. Our code is available at https://github.com/deephoony/plato.

1 Introduction

Context-based offline meta-reinforcement learning adapts from fixed offline datasets, but representation improvements alone do not train the policy on the off-support latents encountered during parameter extrapolation. PLATO synthesizes outward latents for policy training, with theoretical guarantees and improved out-of-support returns across eight MuJoCo task families.

Background

  • The introduction frames offline meta-RL as adapting to new tasks from fixed datasets without further interaction, using a context encoder and shared latent-conditioned policy.This setting is motivated by domains where online interaction is costly or unsafe.Source: §1 · p. 1

Related work

  • Prior COMRL progress has largely improved task representations to separate task identity from behavior-policy artifacts, with transfer beyond the training range implicitly expected.The cited approaches include contrastive structure, information-theoretic objectives, and behavior-policy mutual-information minimization.Source: §1 · pp. 1–2

Motivation

  • On parameter-extrapolation tasks, the encoder produces latents outside the training support, where the policy has never been trained.Improving the encoder alone does not close this gap; a tighter training latent support can widen it.Source: §1 · p. 2

PLATO

  • PLATO assumes task latents lie on a low-dimensional manifold and out-of-support tasks lie farther from the training centroid, then perturbs inferred latents outward.A decoder ensemble rolls out short trajectories at the perturbed latents, and disagreement down-weights uncertain imagined steps in policy updates.Source: §1 · p. 2

Contributions

  • PLATO achieves the highest average out-of-support return against five COMRL baselines on all eight MuJoCo task families while remaining competitive in distribution.Its policy is trained on counterfactual off-support latents using pessimism-weighted imagined rollouts.Source: §1 · p. 2

2 Preliminaries

The preliminaries formalize offline meta-RL task and latent support, then distinguish epistemic uncertainty from irreducible noise. PLATO uses decoder-ensemble predictive variance as a proxy for epistemic uncertainty at off-support latents.

Task and latent setup

  • Out-of-support tasks are parameter extrapolations beyond the training range, and PLATO targets this within-family setting rather than transfer to qualitatively different task families.The training latent support is the support of the encoder’s inferred latent distribution.Source: §2 · p. 2

Uncertainty

  • PLATO needs epistemic uncertainty because it signals where data fail to constrain the model; aleatoric uncertainty is irreducible environment noise shared across training and out-of-support tasks.The epistemic component is mutual information between the next state and decoder parameters conditioned on state, action, and latent.Source: §2 · p. 3

Practical estimation

  • A bootstrapped decoder ensemble approximates the intractable posterior, and under a Gaussian decoder likelihood its predictive variance is a monotone proxy for the mutual-information signal.PLATO uses this variance proxy at perturbed latents in the pessimism weight rather than estimating mutual information directly.Source: §2 · p. 3

3 Method

PLATO combines a task-representation backbone with outward latent perturbations, ensemble rollouts, and pessimism-weighted policy updates. Its geometric assumption motivates which latents to synthesize, while model disagreement controls how strongly imagined transitions influence the policy.

3 Method

  • PLATO adds outward latent perturbation, decoder-ensemble imagined rollouts, and a pessimism-weighted policy update to a representation backbone.Figure 2 presents this training-time pipeline.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Representation backbone

  • The representation backbone can be any context encoder whose latent geometry reflects task identity rather than behavior policy; the paper uses CSRO with a bootstrapped decoder ensemble.The ensemble predicts normalized next-state delta and reward, and its variance supplies the disagreement signal.Source: §3.1 · p. 3

3.2 Manifold hypothesis on the task family

  • The manifold hypothesis assumes a smooth, low-dimensional task-latent manifold whose out-of-support latents lie farther from the training centroid on average.Outwardness is a sufficient condition for the perturbation direction to point toward held-out tasks, not a requirement of PLATO as a whole.Source: §3.2 · p. 4

3.3 Radial outward perturbation

  • The radial perturbation increases each latent’s centroid distance by ασd(di/ d̄)^p, scaling displacement with batch spread and assigning larger moves to peripheral samples.With default p = 2, a latent twice the mean distance moves four times as far; the rule uses batch statistics and no task labels.Source: §3.3 · p. 5

3.4 Imagined rollout and the disagreement signal

  • At each perturbed latent, the decoder-ensemble mean rolls out a short trajectory from a real state, while per-step variance across state-delta and reward channels provides an epistemic-disagreement proxy.The disagreement proxy is monotone in epistemic mutual information and can remain bounded away from zero off-support under Theorem 1’s conditions.Source: §3.4 · p. 5

3.5 Pessimism-weighted policy improvement

  • The policy update uses imagined transitions in proportion to model confidence, while the default critic trains only on real transitions and excludes imagined data from Bellman targets.The pessimism weight decreases with disagreement; the critic choice avoids the bias risk analyzed for mixed-data targets.Source: §3.5 · pp. 5–6

4 Analysis

The analysis links PLATO’s perturbation reach to off-support placement and a value-level generalization bound, and gives conditional guarantees for persistent disagreement. It also shows why mixing imagined data into critic targets can introduce model-error-dependent bias.

4 Analysis

  • The analysis establishes controllable off-support reach, conditional persistence of disagreement, a worst-case critic-bias result, and a value-gap bound governed by perturbation reach.The critic result motivates the real-transition-only default, while the reach controls both off-support displacement and the generalization guarantee.Source: §4 · p. 6

4.1 The perturbation pushes latents off the training support

  • The perturbation moves a latent from distance di to di + ασd(di/ d̄)^p, and it leaves the training support when that added reach exceeds R − di.For peripheral latents with di ≥ d̄, α ≥ (R − di)/σd suffices.Source: §4.1 · p. 6

4.2 Persistence of off-support disagreement

  • Under the stated decoder-Lipschitz and transport-margin assumptions, off-support ensemble disagreement stays bounded away from zero and epistemic mutual information remains positive.The condition is easier to satisfy at later anchors because the remaining parameter-movement tail budget shrinks; it can fail on-support, where training reduces disagreement directly.Source: §4.2 · p. 6

4.3 Bias from imagined data in the critic

  • In the worst case, mixing imagined data into Bellman targets induces bias linear in rollout-model error; with β = 0, the critic error bound has no term linear in εP.The result motivates training the critic on real transitions by default, especially because model error can be large at untrained perturbed latents.Source: §4.3 · p. 7

4.4 Out-of-support generalization gap is bounded by the perturbation reach

  • Under the stated Lipschitz condition on value along the latent manifold, the out-of-support generalization gap is controlled by perturbation reach rather than ambient dimension.Increasing α enlarges both the perturbation reach and the value-level radius covered by the non-trivial guarantee.Source: §4.4 · p. 7

5 Related Work

Related work improves task representations, models uncertainty in imagined transitions, or addresses context shift at adaptation time. PLATO instead intervenes at the policy level with structured off-support latent training.

Context-based offline meta-RL

  • PLATO is encoder-agnostic and intervenes at the policy level, extending UNICORN’s Gaussian latent perturbation with structured outward perturbations, per-step pessimism, and a critic trained on real data.UNICORN [19, Sec. 4.2] explores Gaussian perturbations and decoded rollouts; PLATO develops that idea with these three elements.Source: §5 · p. 8

Model-based offline RL and uncertainty penalties

  • Unlike MOPO, MOReL, and MerPO, PLATO uses imagined transitions only in the policy gradient and generates them at perturbed task latents.The cited methods penalize reward for ensemble disagreement or, for MerPO, mix imagined transitions into the Bellman target.Source: §5 · p. 8

Other directions

  • PLATO adapts latent perturbation to offline meta-RL with structured outward movement, while other approaches address training-to-adaptation context shift at adaptation time.The cited alternatives include in-distribution episode selection, flow-based correction, and reward-free online rollouts.Source: §5 · p. 8

6 Experiments

Experiments compare PLATO with five COMRL baselines across eight MuJoCo task families, measuring held-out out-of-support returns and in-distribution performance. They also examine component ablations and whether decoder disagreement distinguishes in-support from out-of-support tasks.

Experimental setup

  • The evaluation used held-out task-parameter ranges across eight MuJoCo families and compared PLATO with five COMRL baselines under the same datasets and protocol.Returns were reported separately for in-distribution and out-of-support splits, as means ± standard deviations over five seeds.Source: §6 · p. 8

Out-of-support performance (Q1)

  • PLATO achieved the highest average out-of-support return on all eight benchmarks and the lowest seed variance on Ant-Dir, Ant-Goal, Hopper-Mass, and Walker-Friction.The return margins were substantial on Humanoid-Dir and Walker-Friction, and larger on Ant-Dir and Walker-Mass.Source: §6 · pp. 8–9

In-distribution performance (Q2)

  • PLATO remained competitive on in-distribution tasks, with a small, environment-dependent Hopper-Friction trade-off linked to synthetic gradients.The pessimism weight stays close to one on in-support inputs, and the critic is trained only on real data.Source: §6 · p. 9, Table 2 · p. 9

Component ablations (Q3)

  • No-weight and Q-synth underperformed the full algorithm on all six environments, while no-direction underperformed on five; the centroid-outward direction was more consistent across families.Figure 4 compares the default with random-direction, unweighted, and imagined-Bellman-target variants.Source: §6 · p. 9, Figure 4 · p. 10

The off-support disagreement signal (Q4)

  • 13-fold higher disagreement on out-of-support than in-support tasks was measured on Hopper-Friction; the gap persisted across training checkpoints.The gap ended at 27.8 times the final on-support level, and off-support disagreement never fell below 11.5 times that level.Source: §6 · p. 9, Figure 3 · p. 9

Further analyses

  • Further analyses found disagreement correlated with naive-agent task difficulty (rank correlation 0.55 versus −0.02 for latent distance), while out-of-support return was largely insensitive to ensemble size.A smaller ensemble mainly increased variability across seeds.Source: §6 · p. 10
  • On MetaWorld Reach, PLATO achieved the highest out-of-support success rate (0.53 versus 0.43 for the strongest baseline) and return.MetaWorld Reach is a manipulation task outside the locomotion families.Source: §6 · p. 10

7 Discussion

PLATO exposes the policy to off-support latents during training; the discussion reports improved out-of-support returns across eight MuJoCo benchmarks without sacrificing in-distribution performance. The appendices describe its training and testing procedures, the ensemble-variance proxy for epistemic uncertainty, and the geometric condition for moving latents beyond training support.

7 Discussion

  • Across eight MuJoCo benchmarks, PLATO improves out-of-support returns on every environment without sacrificing in-distribution performance.The discussion says gains also carry over to a manipulation task.Source: §7 · p. 10

A Context-based Offline Meta-RL: A Brief History

  • PEARL established the shared context-encoder paradigm; later methods refined encoder objectives, addressed behavior-policy shortcuts, and explored orthogonal extensions [33].Separate approaches use task-conditioned diffusion planning or transformers.Source: A Context-based Offline Meta-RL: A… · p. 13

B Training and Inference Algorithms

  • PLATO adds radial latent perturbations and pessimism-weighted imagined rollouts during meta-training, while keeping critic updates on real transitions.At test time, the encoder and policy follow the standard context-based procedure without rollouts or extra computation.Source: B Training and Inference Algorithms · pp. 14–15

C Ensemble Variance as an Epistemic-MI Proxy

  • Under a Gaussian decoder likelihood, ensemble prediction variance is a monotone proxy for epistemic mutual information.The sample covariance trace converges in probability to the population trace as the ensemble size grows, assuming independent posterior draws and bounded predictions.Source: C Ensemble Variance as an Epistemic-MI… · p. 15

D Proofs

D.1 Proof of Proposition 1 (off-support guarantee)

  • For peripheral latents with di ≥ dbar, α ≥ (R − di)/σd suffices to move zaug outside Ztrain when Ztrain is contained in B(c, R).This follows from the strictly increased centroid distance and the stated perturbation condition.Source: §D.1 · p. 17

D.2 Proof of Theorem 1 (persistence of off-support disagreement)

The proof links off-support disagreement to a conditional margin, while the accompanying sections bound critic bias and out-of-support generalization by model error and perturbation reach. The experiments specify held-out task splits and find the manifold-distance assumption informative in most tested environments, but not all.

D.2 Proof of Theorem 1 (persistence of off-support disagreement)

  • Off-support disagreement stays positive at every later training step when Assumption 3's margin holds, while the result does not quantify its magnitude.The empirical check found it never fell below 11.5 times the final on-support level and ended 27.8 times higher.Source: §D.2 · p. 17

D.3 Proof of Theorem 2 (critic bias under mixed-data Bellman targets)

  • Mixing imagined transitions into Bellman targets can induce worst-case critic bias linear in rollout-model error, whereas PLATO's real-only critic avoids that term.With β = 0, the population critic equals Q*; finite-sample error is O(εdata).Source: §D.3 · pp. 18–19

D.4 Proof of Proposition 2 (out-of-support generalization gap)

  • The out-of-support policy-value gap is bounded by the perturbation reach, with higher-order terms absorbed into O(δ).The proof combines a Lipschitz argument along the latent task manifold with the geometric reach bound.Source: §D.4 · p. 19

E Environment Details

  • Each environment uses 20 in-distribution training tasks and 10 out-of-support evaluation tasks selected from parameter extremes; intermediate grid points are excluded.The benchmarks span eight MuJoCo continuous-control environments, and the table summarizes state/action dimensions and task counts.Source: E Environment Details · p. 19, Table 3 · p. 20

E.1 Reward-shift environments

  • Reward-shift tasks extrapolate beyond training reward parameters, including held-out headings, goals, and target velocities.Ant-Goal extrapolates in both radius and angle; Ant-Dir and Humanoid-Dir hold out front-facing headings.Source: §E.1 · p. 20

E.2 Dynamics-shift environments

  • The four dynamics-shift environments test physics extrapolation at both extremes of the parameter range, outside the central training band.Mass and friction scale factors span 0.44 to 2.25, with training tasks in the 0.66 to 1.52 band.Source: §E.2 · pp. 20–21

E.3 Manipulation environment

  • MetaWorld Reach holds out 10 outer-shell goals beyond the training region; the 10 farthest corner goals are reserved for extrapolation-limit analysis.The training split uses the 20 cells of the inner 3 × 3 × 3 cube.Source: §E.3 · p. 21

E.4 Empirical support for the manifold hypothesis

  • Out-of-support latents are farther from the centroid on Ant-Goal, Cheetah-Vel, and Ant-Dir, but not Walker-Mass; PLATO still leads on Walker-Mass.The authors treat the manifold assumption as a sufficient condition for an informative outward direction, not a required property.Source: §E.4 · p. 21

E.5 Disagreement tracks task difficulty

Across Ant-Dir, Hopper-Friction, and Ant-Goal, decoder-ensemble disagreement tracks the difficulty of tasks for an agent without off-support exposure, whereas latent distance does not. This supports using disagreement to scale PLATO's pessimism weight.

E.5 Disagreement tracks task difficulty

  • On Ant-Dir, disagreement correlates with normalized regret (ρ = 0.55, p = 0.002), while latent distance does not (ρ = −0.02, p = 0.92).Within out-of-support tasks, disagreement correlation is ρ = 0.77 (p = 0.009); the pattern also holds on Hopper-Friction and Ant-Goal.Source: §E.5 · p. 22, Figure 5 · p. 22
  • Disagreement orders held-out Ant-Dir tasks by difficulty, not merely by whether they are out-of-support.Every out-of-support task has higher disagreement than every training task; mean regret is 0.91 versus 0.26.Source: §E.5 · p. 22, Figure 5 · p. 22
  • PLATO's disagreement-based pessimism discounts imagined transitions most where a naively trained agent struggles and relaxes where the decoder generalizes.The reported results support scaling pessimism by disagreement rather than latent distance.Source: §E.5 · p. 22

F Offline Data Collection

The offline datasets use per-task SAC experts and saved trajectory checkpoints, combining near-asymptotic behavior with exploratory transitions. Out-of-support data is reserved for evaluation, while training contexts are sampled independently from in-distribution task buffers.

F Offline Data Collection

  • Each task's offline dataset comes from a separately trained SAC expert, with identical data supplied to PLATO and its baselines.The expert is trained from scratch for 10^6 environment steps on the single-task MDP.Source: F Offline Data Collection · p. 22
  • Saving 19 checkpoints per task, each with 50 rollouts of length 200, yields an offline budget of at most 1.9 × 10^5 transitions per task.Evaluation rollouts are recorded every 5 × 10^4 environment steps over the interval [10^5, 10^6).Source: F Offline Data Collection · pp. 22–23
  • The combined collection mixes late near-asymptotic expert behavior with exploratory transitions from earlier checkpoints.The behavior policy reaches a return close to its locomotion asymptote within roughly 5 × 10^5 steps.Source: F Offline Data Collection · p. 23
  • Out-of-support transitions are loaded only for evaluation and never contribute to a training gradient.Training contexts are drawn independently from in-distribution task buffers.Source: F Offline Data Collection · p. 23

G Implementation Details

PLATO uses shared encoder and actor–critic backbones with a decoder ensemble, while a few PLATO-specific hyperparameters vary by environment. Training costs roughly 5–15% more wall-clock time than CSRO, with decoder rollouts and uncertainty computation accounting for the overhead.

G Implementation Details

  • The encoder uses a permutation-invariant set architecture, and the decoder ensemble has 10 bootstrapped members predicting state deltas and rewards.The encoder and CLUB use 3-layer MLPs; the encoder input differs between reward-shift and dynamics-shift task families.Source: G Implementation Details · p. 23
  • PLATO inherits CSRO's encoder objectives and BRAC actor–critic backbone, keeping that backbone identical across PLATO and CSRO runs.Per-environment settings include perturbation magnitude, pessimism strength, rollout horizon, and warm-up.Source: G Implementation Details · pp. 23–24
  • All eight environments use 200,000 policy-gradient updates, with final metrics averaged across five seeds.Each final-iteration metric averages five evaluation rollouts per task before averaging across seeds.Source: G Implementation Details · p. 24

H Computational Cost

  • Median training time ranges from about 7 hours on Cheetah-Vel to about 30 hours on Humanoid-Dir.Reported per-environment times trim the slowest 10% of seeds to remove contention outliers.Source: H Computational Cost · p. 24
  • PLATO adds roughly 5–15% wall-clock overhead over CSRO from decoder rollouts and per-step uncertainty computation.The overhead does not grow with the number of training tasks or the out-of-support evaluation set.Source: H Computational Cost · p. 25

I Sensitivity Analyses

Across the sensitivity sweeps, PLATO’s returns are generally stable, with perturbation reach and pessimism strength showing the clearest environment-dependent effects. Smaller decoder ensembles largely preserve out-of-support performance, while the learning-curve figures cover in-distribution and out-of-support results across eight MuJoCo benchmarks.

I Sensitivity Analyses

  • Across the four environments, final returns mostly cluster within one seed standard deviation across the swept hyperparameter ranges.The 10K warm-up used in the main experiments is never far from the best value in the sweep.Source: I Sensitivity Analyses · pp. 25, 27
  • Ant-Goal performs substantially better with α = 1.5 than with α = 0.7 or α = 0.1, while the other environments are less sensitive to perturbation reach.Held-out Ant-Goal tasks are displaced both radially and angularly, so smaller perturbations miss their region; the other three environments hold out tasks along a single axis.Source: I Sensitivity Analyses · pp. 25–26, Figure 6 · p. 26
  • λpess = 5 gives the most consistent returns across the four environments; both λpess = 0 and λpess = 10 perform slightly worse.Without pessimism, the policy treats imagined transitions as fully reliable; excessive pessimism suppresses training signal even at moderate disagreement.Source: Figure 6 · p. 26, I Sensitivity Analyses · p. 26
  • Warm-up effects vary by environment: 30K is lowest on Ant-Goal, 10K is best on Walker-Friction, and no setting dominates across environments.Cheetah-Vel and Hopper-Mass curves largely overlap, and the 10K main-experiment setting remains near the best sweep value.Source: Figure 6 · p. 26, I Sensitivity Analyses · p. 26
  • Reducing ensemble size largely preserves out-of-support performance, but smaller ensembles increase variability across random subsets and seeds.With random subsets, AUROC changes by at most 0.02 between M′ = 10 and M′ = 2 in every environment; retrained smaller ensembles stay within 6% of M = 10 on Cheetah-Vel and Hopper-Mass, while M = 5 matches M = 10 on Walker-Friction.Source: I Sensitivity Analyses · pp. 26–27, Table 6 · p. 27

J Learning Curves

  • Figures 7 and 8 show normalized-return learning curves for the eight MuJoCo benchmarks, in-distribution and out-of-support, averaged over five seeds.Shaded regions indicate one standard deviation.Source: Figure 7 · p. 27, J Learning Curves · p. 27

K Additional Domain: MetaWorld Reach

On MetaWorld Reach, PLATO leads the out-of-support methods on success rate and return while remaining competitive on training tasks. Its comparison with MerPO also shows an out-of-support advantage over a baseline that adapts with test-task gradient steps.

K Additional Domain: MetaWorld Reach

  • On MetaWorld out-of-support tasks, PLATO attains the highest success rate (0.53 ± 0.09, against 0.43 ± 0.13 for FOCAL) and return (1575±39, against 1506±78 for CSRO).It leads for most of training, not only at the final iteration.Source: Figure 9 · p. 28, K Additional Domain: MetaWorld Reach · p. 28
  • On MetaWorld training tasks, PLATO matches the best return (1692 ± 28, against 1693 ± 90 for FOCAL), while its success rate is close to the strongest baselines.At goal-box corners about 1.9× the training radius, every method’s success rate falls to at most 0.10, although PLATO has the highest return there (1108, against 1083 for FOCAL).Source: K Additional Domain: MetaWorld Reach · p. 28

L Comparison with MerPO

  • Against MerPO, PLATO leads out-of-support by more than two standard deviations on Ant-Dir, Humanoid-Dir, and Walker-Friction, and by more than one on Hopper-Friction and Walker-Mass.The methods are within one standard deviation on Ant-Goal and Hopper-Mass; Cheetah-Vel is omitted because MerPO did not train stably.Source: Table 7 · p. 29, L Comparison with MerPO · p. 29
  • PLATO’s out-of-support advantage over MerPO holds against a baseline that takes gradient steps on each test task, whereas PLATO takes no test-time gradient step.MerPO’s within-task policy optimization adapts using each new task’s offline data.Source: L Comparison with MerPO · p. 29

M Limitations

PLATO’s guarantees depend on task geometry and extend only over a bounded extrapolation range. The paper also identifies practical trade-offs and cautions that improved generalization does not remove risks beyond the method’s assumptions.

M Limitations

  • The centroid-outward assumption fails for annulus geometry, where held-out tasks lie inside the outer training envelope and PLATO’s guarantees do not transfer.On this Ant-Goal construction, a random perturbation direction performs better on inner held-out goals, and the disagreement signal does not flag the geometry inversion.Source: M Limitations · p. 29
  • Perturbation reach and rollout horizon trade coverage against rollout drift, and extrapolation remains limited: every method’s success rate falls to at most 0.10 at MetaWorld corners about 1.9× the training radius.The centroid and spread estimates inherit finite-mini-batch noise; the main benchmarks are MuJoCo locomotion plus one manipulation task, with vision-based domains and cross-family transfer left for future work.Source: M Limitations · p. 30

N Broader Impact

  • The off-support guarantee is local to the perturbation reach, not the ambient task space; safety-critical deployment therefore calls for runtime safeguards.The paper notes that policies may fail more confidently when situations lie beyond the method’s assumptions.Source: N Broader Impact · p. 30

Related papers

Loading 2610.15428v2…