PaperlayerView exact revision on arXiv

Source-linked AI summary

ProxPI: Proximal Prior Injection for Sampling-Based MPC under Learned-Prior Mismatch

Euncheol Im, Myotaeg Lim, Yisoo Lee

arXiv:2609.00941v1cs.ROeess.SY

TL;DR

Policy-centered warm-start can repeatedly concentrate MPPI exploration around a mismatched learned prior, preventing recovery toward the task optimum. ProxPI retains nominal-centered sampling and adds the prior as a soft proximal cost; it matches aligned-prior schemes and recovers vanilla MPPI-level performance under mismatch across simulations and real-robot experiments.

  • Problem

    Policy-centered warm-start can restrict exploration around an out-of-distribution prior and repeatedly overwrite MPPI corrections, while the effect of this persistent re-centering is insufficiently characterized.

  • Method

    ProxPI keeps MPPI sampling centered at the evolving nominal sequence and incorporates the learned prior through a proximal trajectory-cost term.

  • Results

    ProxPI matches other prior-injection schemes under aligned priors and recovers vanilla MPPI-level performance under mismatch across five simulated platforms and a real FR3 task.

  • Takeaways & Limitations

    Nominal-centered updates retain corrective shifts across replans, whereas persistent policy-centered re-centering discards them; increasing rollout budget does not reliably remove this failure.

Abstract

from arXiv · show

Combining learned policies with model predictive control can leverage learned task priors while retaining online adaptation to new objectives and constraints, but performance degrades when the policy is out of distribution. In policy-guided model predictive path integral (MPPI) control, a policy-centered warm-start approach centers the sampling distribution on the policy output. When the prior is mismatched, centering the sampling distribution on the policy output restricts exploration around an unsuitable solution and prevents recovery toward the task optimum. We propose Proximal Prior Injection (ProxPI), which retains nominal-centered MPPI sampling and incorporates the policy through a soft proximity cost. This matches the in-distribution performance of existing prior-injection schemes while enabling the optimizer to escape an inaccurate policy and recover vanilla MPPI-level performance. We theoretically show that re-centering on the prior discards the optimizer's correction at every update, whereas nominal-centered sampling retains it and converges to a solution set by both the task cost and the prior, and that this failure is not removed by a larger rollout budget. Simulations and real-robot experiments demonstrate robust performance under both in-distribution and out-of-distribution tasks.

I. INTRODUCTION

Learned policies provide task-relevant structure, while sampling-based MPC re-optimizes deployment-time objectives and constraints. ProxPI keeps the optimizer's nominal-centered exploration while using the prior as an escapable preference, addressing persistent re-centering under prior mismatch.

  • Motivation: Learned policies efficiently produce task-relevant actions but do not explicitly re-optimize newly specified deployment objectives or constraints.Sampling-based MPPI instead evaluates deployment-time trajectory costs through model rollouts without requiring analytic dynamics derivatives.
  • Prior mismatch: When a learned proposal is mismatched, policy-centered warm-start repeatedly samples around a high-cost region and overwrites corrective nominal shifts.This structural re-centering can leave task-relevant regions with negligible coverage under finite rollout budgets.
  • ProxPI: ProxPI keeps sampling centered at the evolving MPPI nominal and injects the learned prior through a soft proximal trajectory cost.A moderate proximal weight biases aligned optimization toward promising sequences while allowing escape under mismatch; zero weight recovers vanilla MPPI.
  • Theory: Theoretical analysis shows nominal-centered updates retain task-driven corrections, whereas policy-centered warm-start repeatedly discards them.The analysis also separates persistent re-centering from finite-sample weight concentration and shows that larger rollout budgets do not remove the re-centering failure.
  • Validation: Simulations and multi-platform experiments show ProxPI matches aligned-prior schemes while remaining robust to prior mismatch.The evaluated platforms include a Summit XLS mobile robot, planar arm, FR3 manipulator, Go2 quadruped, and G1 humanoid.

A. Stochastic Optimal Control

MPPI formulates stochastic optimal control through sampled perturbed control sequences and trajectory-cost evaluation. Its information-theoretic relaxation identifies a Gibbs target whose mean is estimated using cost-based importance weighting.

  • Stochastic optimal control: The decision variable is a horizon-length nominal control sequence that MPPI maintains as its current optimal-control estimate.The nominal sequence also centers the Gaussian sampling distribution used to generate candidate inputs.
  • Stochastic optimal control: MPPI perturbs the nominal sequence with zero-mean Gaussian noise to generate rollout control sequences.Each rollout is then evaluated using a trajectory cost with running and terminal terms.
  • Stochastic optimal control: The stochastic control objective minimizes expected trajectory cost over the rollout distribution.The trajectory cost can include tracking error, control effort, and constraint penalties.
  • Information-theoretic solution: MPPI relaxes the expected-cost problem into a free-energy objective combining expected trajectory cost with KL divergence from the nominal-induced base distribution.The minimizer is a Gibbs distribution parameterized by temperature and the base sampling distribution.
  • Information-theoretic solution: The MPPI update estimates the Gibbs distribution's mean using a self-normalized importance-weighted average over sampled rollouts.Because the proposal equals the base distribution, the importance weights reduce to functions of trajectory cost.

C. Quality of the Sample-Based Estimate

Finite rollout budgets make MPPI quality depend on proposal coverage of the optimal distribution, with weight concentration producing importance-sampling degeneracy. ProxPI changes the injection site by retaining nominal-centered sampling and adding a prior penalty to the target cost.

  • Sample-based estimate: Under finite rollout budgets, poor proposal coverage of the optimal distribution concentrates importance weights on few samples and makes the MPPI estimate noisy.This degradation is measured by the chi-squared divergence between target and sampling distributions.
  • Sample-based estimate: Effective sample size equals K/(1 + χ2), reaching K when proposal and target distributions coincide.It represents the number of rollouts contributing as if equally weighted and falls to 1 when one rollout carries all weight.
  • Repeated updates: Nominal-centered updates accumulate task-driven corrections toward a fixed point determined jointly by task cost and prior, unlike policy-centered re-centering.The analysis treats finite-sample weight concentration as a separate mechanism from persistent re-centering.
  • Prior injection: ProxPI leaves q = N(U, Σ) unchanged and adds a quadratic residual penalizing deviation from the learned prior.Warm-start instead relocates the sampling distribution so rollouts perturb the prior rather than the nominal sequence.
  • Prior injection: ProxPI and warm-start both inject a Gaussian prior centered at Up, but ProxPI places it in the target while warm-start places it in the sampling distribution.ProxPI's prior covariance is λ^2/α I, whereas warm-start uses the sampling covariance Σ.

B. Local Nominal Dynamics

The analysis models repeated MPPI updates around a local task-cost minimizer and distinguishes nominal-centered sampling from policy-centered warm-start. Nominal-centered updates retain corrections and converge geometrically, whereas warm-start repeatedly resets to the policy-centered update.

  • B. Local Nominal Dynamics: The local model fixes the task minimizer, curvature, prior, and sampling covariance to isolate repeated-update behavior.The task cost is locally approximated, with α = 0 recovering the task cost optimized by vanilla MPPI and warm-start.
  • B. Local Nominal Dynamics: The proposal center determines the update dynamics: nominal-centered sampling uses the previous nominal, while warm-start uses the policy sequence at every update.Both schemes are expressed through the same MPPI update, but they supply different proposal centers.
  • B. Local Nominal Dynamics: Under nominal-centered sampling, the update matrix has spectral radius below one and the nominal converges geometrically to the fixed point U †.The fixed point is determined by the task and proximal objective under the local approximation.
  • B. Local Nominal Dynamics: Policy-centered warm-start returns m0(Up) at every update, independently of preceding updates, so estimated corrections are not carried forward.The one-step MPPI correction still moves toward the task minimizer, but re-centering removes its accumulation across updates.

C. Finite-Sample Weight Degeneracy

Finite rollout budgets introduce weight degeneracy when proposal and target distributions are displaced, while proximal weighting creates a separate convergence–bias trade-off. Persistent re-centering remains an infinite-sample failure mechanism, so larger rollout budgets cannot remove it.

  • C. Finite-Sample Weight Degeneracy: Finite-K MPPI updates contain an importance-sampling error whose reliability depends on the proposal center used by each scheme.ProxPI and vanilla MPPI center proposals at the current nominal, whereas warm-start centers them at the prior sequence.
  • C. Finite-Sample Weight Degeneracy: The χ2 divergence measures weight degeneracy caused by proposal–target displacement, but evaluates reliability of each scheme’s own update rather than task quality.The covariance mismatch is finite because Cα ⪯ Σ under H ⪰ 0 and α ≥ 0.
  • C. Finite-Sample Weight Degeneracy: For warm-start, the mean-displacement contribution remains set by U∗−Up, whereas ProxPI’s corresponding contribution vanishes as its nominal approaches the fixed point.Warm-start repeatedly resets the proposal center to Up, while ProxPI uses the evolving nominal.
  • C. Finite-Sample Weight Degeneracy: Persistent re-centering discards corrections even in the infinite-sample limit, so increasing the rollout budget does not remove this failure mechanism.Weight concentration and persistent re-centering are identified as distinct mechanisms.
  • D. Proximal-Weight Trade-off: Increasing α lowers the surviving task-error fraction and raises the retained prior displacement, trading faster local contraction for greater prior bias.Excessive proximal guidance can therefore reduce robustness under mismatch, especially together with weight concentration.

IV. EXPERIMENTS

The experiments compare ProxPI with six prior-injection baselines and vanilla MPPI across five simulated platforms, with additional real-FR3 validation. Evaluation spans environment mismatch, behavior mismatch, and platform-specific task-progress metrics.

  • IV. EXPERIMENTS: The study evaluates ProxPI, six prior-injection baselines, and vanilla MPPI on five simulated robotic platforms, plus warm-start and ProxPI on a real FR3 arm.Simulated results use N = 50 runs unless otherwise stated and report task-progress scores as mean ± standard deviation.
  • IV. EXPERIMENTS: Environment mismatch introduces unseen obstacles in 2D navigation, 2D reaching, and simulated FR3 tasks.The in-distribution setting retains the training environment, while OOD evaluation blocks the direct route or otherwise changes the environment.
  • IV. EXPERIMENTS: Behavior mismatch commands Go2 rearing and G1 sustained squatting while retaining velocity-tracking locomotion priors trained for different objectives.Both commanded motions lie outside their respective prior objectives.
  • IV. EXPERIMENTS: The compared methods differ in how learned priors enter sampling and objectives, with the study isolating injection mechanisms across schemes.For GPC-CEM, prior samples are drawn from the same learned prior as other methods to isolate the injection mechanism.
  • C. Task-Progress Metrics: Simulated progress scores are normalized per platform, with p = 1 indicating a target reached and maintained and p = 0 indicating no progress or failure.Unless collision occurs, components are averaged over the final 25% of each episode to measure sustained completion.

1) Collision-Free Goal Reaching:

The collision-free goal-reaching evaluation measures sustained target progress across navigation, reaching, and higher-dimensional tasks. Under OOD mismatch, warm-start fails while vanilla MPPI and ProxPI complete the evaluated tasks.

  • 1) Collision-Free Goal Reaching:: Under OOD mismatch across five robots, warm-start fails all tasks, whereas vanilla MPPI and ProxPI complete them.The figure marks successes and failures and reports failure reasons in unsuccessful panels.
  • 1) Collision-Free Goal Reaching:: Motion toward the target increases the progress score, while obstacle contact is treated as task failure.The metric is designed to reflect task progress rather than merely final configuration.
  • 1) Collision-Free Goal Reaching:: FR3 progress requires both end-effector position and orientation completion, with the minimum preventing either component alone from receiving full credit.This enforces simultaneous pose completion.
  • 2) Go2 Bipedal Rearing:: The evaluation also requires simultaneous Go2 progress in base height and trunk pitch.These task-specific metrics extend goal-reaching assessment beyond planar position error.
  • 1) Collision-Free Goal Reaching:: Each controller is evaluated on whether it reaches and maintains the newly commanded target.Sustained target maintenance is part of the reported task outcome.

D. In-Distribution: Performance under an Aligned Prior

Under an aligned prior, all seven injection schemes achieve near-optimal performance, while under prior mismatch ProxPI preserves task progress across simulated platforms and avoids the failures of prior-centered approaches.

  • In-Distribution: Performance under an Aligned Prior: All seven injection schemes reach near-optimal progress on the two in-distribution tasks, with scheme ranges of 0.009 for navigation and 0.007 for reaching.At this operating budget, the injection site does not change in-distribution performance.
  • Out-of-Distribution Robustness: ProxPI achieves 0.989 navigation and 0.983 planar-reaching progress under OOD mismatch, matching vanilla MPPI while warm-start collides and fails.ProxPI allows the nominal sequence to move away from the mismatched prior and produce collision-free trajectories.
  • Controlled Prior Interpolation: As prior mismatch increases, warm-start transitions from successful completion to complete failure, while ProxPI succeeds on all 50 tested seeds at every interpolation coefficient.On reaching, warm-start fails from ρ = 0.3 even while normalized ESS remains between 0.259 and 0.290 through ρ = 0.7.
  • Controlled Prior Interpolation: At ρ = 0.3, the interpolated prior has higher task cost than the carried nominal on 94% of navigation replans and 71% of reaching replans.This indicates that re-centering can replace a lower-task-cost nominal before ESS collapses.
  • Out-of-Distribution Robustness: On higher-DoF tasks, ProxPI reaches 0.953, 0.895, and 0.952 on FR3, Go2, and G1, respectively, while warm-start obtains zero progress on every task.ProxPI matches vanilla MPPI on FR3 and G1 and exceeds it by 0.136 on Go2.
  • Out-of-Distribution Robustness: Across five simulated platforms, warm-start and Blend collapse to near-zero progress, whereas ProxPI recovers vanilla MPPI-level performance under environment and behavior mismatch.The consistency is associated with displacing every rollout away from the nominal rather than with a particular platform, task, or scheme.

2) Real-World Experiment:

On a real FR3 arm, ProxPI adapts to an out-of-distribution target switch while warm-start remains near the training target. Sensitivity analyses further show that larger budgets do not reliably resolve warm-start failure and that excessive proximal weighting can collapse ProxPI.

  • Real-World Experiment: After the target switches at t=8.3 s, ProxPI reaches the new FR3 target by t=14.3 s, whereas warm-start remains near the training target and fails.Both methods first reach the in-distribution target at t=8 s.
  • Parameter Sensitivity: Increasing rollout budget leaves warm-start near zero on all three OOD tasks even at the largest budget, while increasing K improves vanilla MPPI and ProxPI.Additional samples around the mismatched prior do not resolve warm-start failure.
  • Parameter Sensitivity: Increasing σ/σ0 lets warm-start escape the prior only at extreme noise on 2D reaching, not on 2D navigation or FR3.Excessive noise also degrades vanilla MPPI and ProxPI on FR3, so broader sampling is not reliable across tasks.
  • Parameter Sensitivity: ProxPI maintains high task progress for small to moderate α but degrades and eventually collapses as the prior is weighted too strongly.The collapse occurs at smaller α on FR3 than on the two low-dimensional tasks.

G. Rollout Budget across Injection Schemes

The rollout-budget sweep shows that ProxPI preserves nominal-centered performance under mismatch, while schemes that displace every rollout from the nominal are not rescued by increasing K. Weight-concentration analysis links this difference to the effective rollout budget available to each method.

  • Budget sweep: Increasing K does not rescue Warm-start or Blend, which remain at zero progress across every tested budget.The result rules out insufficient sample budget as the explanation for their failure.
  • Budget sweep: ProxPI stays within 0.003 of vanilla MPPI at every budget and exceeds it by 0.136 to 0.221 on Go2 for K ≥16.
  • Budget sweep: On Go2, the gap between ProxPI and Mixture falls from 0.243 at K = 8 to around zero at K = 128.
  • Effective sample size: At OOD points, Warm-start and Blend have near-floor effective sample sizes, while ProxPI retains 21.1% on Go2 and 19.1% on G1.The corresponding single-rollout floors are 0.78% on Go2 and 1.56% on G1.
  • Interpretation: The study attributes the robustness distinction to retaining MPPI corrections across replans rather than repeatedly discarding them through policy-centered recentering.

APPENDIX A LOCAL GIBBS TARGET AND FIXED POINT

Under the local Gaussian model, substituting the model into the Gibbs target yields explicit precision and linear terms. For nominal-centered sampling, the fixed point satisfies a linear balance between task and prior terms and is contractive under the stated positive-definiteness condition.

  • Local Gibbs target: Substitution into the Gibbs target produces precision Σ^-1 + (H + 2αI)/λ and linear coefficient Σ^-1 Ū + λ^-1HU* + 2αλ^-1Up.
  • Fixed point: The nominal-centered fixed point satisfies (H + 2αI)U† = HU* + 2αUp after cancelling Σ^-1U†.
  • Convergence: Because Aα is symmetric positive definite, all its eigenvalues lie strictly between zero and one, giving ρ(Aα) < 1.
  • Gaussian validity: The Gaussian derivation requires Cα ⪯ Σ so that the required Gaussian integral is finite.

APPENDIX D EXPERIMENTAL DETAILS

The experiments use platform-specific sampling scales and implementation choices for higher-DoF systems, with a real FR3 prior trained on a single fixed target and evaluated without target input.

  • Sampling settings: FR3 joint sampling standard deviations are 0.05 for joints 1–4 and 0.04 for joints 5–7.
  • Sampling settings: G1 sampling deviations range from 0.010 to 0.030 by joint group, while Go2 uses a uniform 0.10.A uniform G1 value destabilizes the arms, and Go2 and G1 hold policy outputs constant over the MPPI horizon.
  • Real-robot prior: The real FR3 ACT prior predicts a 40-step joint-position chunk from joint positions and velocities, using demonstrations for one fixed end-effector target and no target input.
  • Compute: Simulations run on a 20-core Intel Core Ultra 7 265KF CPU, while real experiments use an Intel NUC 13 with a Core i7 processor and 32 GB RAM.
Loading 2609.00941v1…