PaperlayerView exact revision on arXiv

Source-linked AI summary

Accelerating Reinforcement Learning via MPC Solver-Gradient Guidance for Weights-varying MPC

Baha Zarrouki, Arslan Thobani, Jasper Hoffmann, Mattia Piccinini, Rudolf Reiter, Felix Jahncke, Sébastien Gros, Davide Scaramuzza, Johannes Betz

arXiv:2609.01061v1cs.ROcs.LGeess.SY

TL;DR

Fixed MPC weights can be suboptimal across changing conditions, while online adaptation is difficult because policy parameters influence long-horizon behavior through implicit constrained-MPC solutions. The paper introduces SG-RL, which adds bounded solver-gradient guidance to RL while retaining sampled closed-loop return as the objective; on full-scale racing platforms, it improves sample efficiency and return under model mismatch.

  • Problem

    Online MPC weight adaptation must handle changing performance trade-offs, but learning is difficult because constrained solver solutions create long-horizon, nonlinear dependencies on policy parameters.

    Source: §I · p. 1

  • Method

    SG-RL combines environment-based policy gradients with bounded solver-gradient guidance inside PPO while preserving realized closed-loop return as the optimization objective.

    Source: B. Early-Training Guidance · p. 10, §X · p. 17

  • Results

    SG-RL matched PPO’s best return with up to 70.6% fewer training samples, outperformed GB-PL by at least 54% in return, and generalized zero-shot to unseen race tracks.

    Source: §X · p. 17

  • Takeaways & Limitations

    Environment-based policy gradients and model-based solver guidance are complementary signals for learning online MPC weight-adaptation policies.

    Source: §X · p. 17

    GB-PL’s solver-derived gradient can systematically deviate from the true closed-loop improvement direction when the predictive model is inaccurate.

    Source: C. GB-PL Algorithm Limitations · p. 6

Abstract

from arXiv · show

In Model Predictive Control (MPC), cost-function weights shape closed-loop behavior, yet changing conditions often make fixed parametrizations suboptimal and motivate context-dependent online adaptation. Learning such policies is difficult because behavior depends implicitly on numerical MPC solutions, producing nonlinear, potentially nonsmooth, long-horizon dependencies on policy parameters. This creates a bias-variance tradeoff: Reinforcement Learning (RL) optimizes realized closed-loop return from environment samples but is sample-inefficient, whereas Gradient-Based Policy Learning (GB-PL) uses low-variance solver gradients from differentiable MPC to optimize surrogate losses on predicted trajectories but can be biased under model mismatch. We propose Solver-Gradient Guided Reinforcement Learning (SG-RL), a solver-sensitivity augmentation for RL-based online MPC cost-weight adaptation. SG-RL keeps sampled closed-loop return as the objective and uses bounded solver-derived gradients as auxiliary guidance to improve stability and sample efficiency. We instantiate SG-RL in Proximal Policy Optimization (PPO) with four modular algorithms that inject solver-gradient guidance into actor-update scaling, policy loss, advantage estimation, and value-function learning. On two full-scale autonomous racing platforms with intentional model mismatch, SG-RL reaches PPO's best closed-loop return with up to 70.6% fewer samples, outperforms GB-PL baselines by at least 54% in closed-loop return, and generalizes zero-shot to unseen environments.

I. INTRODUCTION

Fixed MPC cost weights can be inadequate across changing operating conditions, while learning online weight-adaptation policies is difficult because constrained MPC solutions create indirect, nonlinear, potentially nonsmooth, long-horizon dependencies. SG-RL combines sampled closed-loop policy gradients with bounded solver-gradient guidance, preserving return optimization while improving learning efficiency.

Motivation

  • Weights-varying MPC adapts cost weights online to preserve context-dependent trade-offs while keeping the underlying model and constraints unchanged.Different racetrack segments can require different performance trade-offs near handling limits.Source: §I · p. 1

Learning challenge

  • Learning online weight-adaptation policies is challenging because policy parameters affect returns through repeated constrained MPC solutions and closed-loop dynamics.This produces long-horizon credit assignment and implicit, nonlinear, potentially nonsmooth mappings.Source: §I · p. 1

Learning signals

  • Environment-based gradients reflect realized closed-loop objectives but can have high variance, whereas solver gradients offer lower-variance local guidance but inherit predictive-model bias.The complementary properties motivate combining both signals within one learning framework.Source: §I · p. 2

SG-RL

  • SG-RL uses solver-gradient guidance as an auxiliary signal for RL while continuing to optimize realized closed-loop return.This distinguishes it from approaches that optimize model-based surrogate objectives using solver sensitivities.Source: §I · p. 2
  • SG-RL is instantiated in PPO through four modular integrations: actor-update scaling, auxiliary policy learning, advantage estimation, and critic learning.The variants are evaluated separately to compare different forms of solver guidance.Source: §2 · p. 3

Validation

  • All SG-RL variants outperform PPO in peak closed-loop performance, reach PPO’s best return with up to 70.6% fewer samples, and outperform GB-PL by at least 54% in return.Evaluation uses two full-scale autonomous-racing simulation platforms under intentional model mismatch and includes zero-shot generalization.Source: §2 · p. 3

II. PROBLEM FORMULATION

The formulation uses a hierarchical weights-varying NMPC architecture: a high-level policy selects cost weights online, and a lower-level constrained NMPC solves and applies the first control input. Training optimizes long-horizon realized closed-loop performance, whose dependence on policy parameters creates a difficult outer optimization problem.

II. PROBLEM FORMULATION

  • Weights-varying NMPC places a high-level learning policy above a low-level receding-horizon controller that solves the constrained OCP at each sampling instant.The policy adapts cost weights along the task domain while the controller computes the applied input.Source: §II · p. 3

A. Weights-varying Nonlinear MPC

  • Online weight adaptation addresses changing operating regimes, disturbances, uncertainty, and task difficulty that make fixed tuning conservative or overly aggressive.Autonomous racing combines straights, braking zones, corners, and exits with different control priorities.Source: 1) Motivation: Why Adapt Cost Weights? · p. 3

2) Weights-varying NMPC OCP Formulation:

  • The NMPC OCP uses dynamics, inequality constraints, and stage and terminal costs parameterized by θ_t, while keeping the prediction model fixed.The weight vector is fixed over each prediction horizon but may change between sampling instants.Source: 2) Weights-varying NMPC OCP Formulation · p. 3

3) Learning-based Cost-Weight Adaptation Policies:

  • A stochastic policy πϕ maps observation o_t to a distribution over NMPC weight vectors, with θ_t serving as the applied action.The policy runs during both training and deployment; only parameter updates distinguish the phases.Source: 3) Learning-based Cost-Weight… · p. 3
  • At each time step, NMPC solves using the current state estimate and θ_t, then applies the first optimal control input in receding-horizon fashion.This couples high-level weight selection to the lower-level controller’s numerical solution.Source: 3) Learning-based Cost-Weight… · p. 3

B. Problem Statement: Learning Policies for Weights-varying NMPC

  • The upper-level objective is to optimize long-horizon closed-loop performance using instantaneous loss evaluated on the realized state and applied control.The performance loss may include collisions, failure-to-finish, or rapid-weight-variation penalties that differ from the internal NMPC cost.Source: B. Problem Statement: Learning… · pp. 3–4
  • Policy learning is difficult because decisions affect future states, observations, solver calls, and returns indirectly through closed-loop interaction with the true plant.The resulting outer problem has long-horizon credit assignment around repeated constrained NMPC solves.Source: B. Problem Statement: Learning… · p. 4
  • The paper develops differentiable-optimization and reinforcement-learning baselines before introducing SG-RL to combine their strengths.The approaches target the upper-level optimization problem for the weight-selection policy.Source: B. Problem Statement: Learning… · p. 4

III. DESIGNING A NEURAL POLICY FOR WEIGHTS-VARYING NMPC

The neural policy maps history-aware observations to bounded, context-dependent NMPC cost weights, allowing reactive, anticipatory, and mismatch-adaptive behavior while preserving valid cost parameters.

Policy architecture

  • The policy maps high-dimensional observations to cost weights for the next NMPC planning horizon, enabling online adaptation when task requirements and disturbances change.The hierarchical design leaves constraints and low-level dynamics to NMPC while the neural policy shapes broader behavioral objectives.Source: §III · p. 4

History-aware inputs

  • The observation combines current measurable context, reference preview, and recent performance history to support reactive, proactive, and adaptive weight changes.Reference preview anticipates braking, curvature, or acceleration events, while performance history helps infer persistent disturbances, model mismatch, or actuator transients.Source: A. Policy Architecture: History-Aware… · p. 4

Constrained outputs

  • Policy outputs are restricted to strictly positive, bounded ranges to preserve conditioning, positive-definiteness for diagonal costs, and numerical stability.Unconstrained logits are mapped into domain-specified limits using a differentiable scaled-tanh parameterization.Source: B. Action Space: Constrained Cost… · p. 4
  • The bounded mapping regularizes the search space and avoids hard clipping, but attenuates gradients near parameter boundaries.The constraints confine optimization to a bounded hyper-rectangle and remove irrelevant parameter regions.Source: B. Action Space: Constrained Cost… · p. 4

Architecture comparison

  • The proposed architecture uniquely combines reactivity, adaptability, and proactive anticipation within a constrained action space.This comparison is stated in the policy-architecture table caption.Source: Table I · p. 5

B. Gradient Propagation and Policy Learning

GB-PL differentiates a smooth surrogate loss through the NMPC solver and policy, while PPO learns from realized closed-loop returns; their trade-offs motivate combining solver guidance with reinforcement learning.

Gradient propagation

  • GB-PL backpropagates a differentiable surrogate loss evaluated on the NMPC-predicted trajectory through solver sensitivities and the neural policy.The chain rule combines loss gradients, solver sensitivity, and the policy Jacobian to obtain a weight-space solver gradient and update policy parameters.Source: B. Gradient Propagation and Policy… · p. 5, §4 · p. 5

GB-PL limitations

  • GB-PL is restricted to objectives with smooth trajectory-based surrogates, excluding or approximating genuinely nonsmooth and sparse event-based terms.Examples include “Lap completed” and “Collision” objectives.Source: B. Gradient Propagation and Policy… · p. 5, C. GB-PL Algorithm Limitations · p. 6
  • GB-PL performs local descent without explicit exploration, making training sensitive to initialization and the surrogate landscape.This limits its ability to address the myopic nature of local gradient descent.Source: C. GB-PL Algorithm Limitations · p. 6
  • GB-PL derives sensitivities from open-loop model predictions, so inaccurate predictive models can make its gradient deviate from the true closed-loop improvement direction.The method primarily tunes MPC costs to match task loss under the solver’s predictive model rather than directly optimizing realized plant behavior.Source: C. GB-PL Algorithm Limitations · p. 6

RL baseline

  • PPO treats the plant and NMPC controller as the closed-loop environment and updates its stochastic weight policy from sampled returns.The reward is evaluated on realized closed-loop states and applied inputs, not the internal NMPC cost or predicted trajectory.Source: §V · p. 6

Learning-signal trade-off

  • PPO’s environment-based signal reflects exploration, stochasticity, and model mismatch, complementing the lower-variance but model-dependent solver gradients.The comparison uses aligned task-level rewards so differences reflect learning signals and update mechanisms rather than different objectives.Source: §V · p. 6

VI. SOLVER-GRADIENT GUIDED REINFORCEMENT LEARNING

SG-RL injects bounded NMPC solver-gradient guidance into PPO while retaining sampled closed-loop learning. Four variants place this guidance in distinct PPO components to compare their effects.

  • SG-RL injects the model-based NMPC solver loss gradient gSG,t into selected components of PPO.Source: §VI · p. 6
  • The solver signal is normalized so ∥gSG,t∥2 ≤1, making its direction comparable across contexts while method-specific coefficients control update magnitude.When sensitivities are unavailable, SG-RL sets gSG,t = 0 and recovers standard PPO behavior.Source: §VI · p. 7
  • Four hybrid methods inject solver guidance through actor-update scaling, performance-gated policy loss, advantage shaping, and critic augmentation.The variants isolate different PPO injection points: SG-SCA, SG-LOS, SG-ADV, and SG-CRT.Source: §VI · p. 7

Adaptive PPO Update Scaling (SG-SCA)

  • SG-SCA scales the PPO update magnitude according to alignment between solver and PPO loss gradients without changing the PPO update direction.Its scaling factor is bounded to prevent excessive steps and cannot reverse the PPO direction.Source: A. Method 1: Adaptive PPO Update… · pp. 7–8

B. Method 2: Performance-Gated Loss (SG-LOS)

SG-LOS selectively supplements PPO with a solver-guided correction when sampled transitions underperform, while avoiding uniform reliance on a local model-based surrogate.

  • SG-LOS adds an explicit target-following loss to PPO when the RL agent performs poorly.Unlike SG-SCA, it adds a loss term rather than rescaling the PPO descent step.Source: B. Method 2: Performance-Gated Loss… · p. 8
  • The method avoids globally regularizing PPO with a local, open-loop, model-based surrogate whose reliability may vary across samples.Solver guidance is instead used to construct a local correction target for selected transitions.Source: B. Method 2: Performance-Gated Loss… · p. 8
  • SG-LOS constructs a target weight configuration by taking a fixed step ηguide along the normalized solver descent direction −gSG,t.Source: B. Method 2: Performance-Gated Loss… · p. 8
  • The current policy mean is penalized for deviating from the solver-suggested target through a gated guide loss.Source: B. Method 2: Performance-Gated Loss… · p. 8
  • Negative-advantage samples receive corrective solver guidance, whereas positive-advantage samples reduce to standard PPO.Transitions with negligible solver-gradient norm are masked out, and the gate activates more often early in training when underperforming transitions are common.Source: B. Method 2: Performance-Gated Loss… · p. 8

C. Method 3: Advantage Shaping (SG-ADV)

SG-ADV incorporates solver-gradient consistency into PPO’s advantage estimates, rewarding policy-mean displacements aligned with local solver descent while leaving the critic architecture unchanged.

Augmented Critic (SG-CRT)

  • SG-CRT instead augments the critic with a residual correction from a normalized solver-gradient encoder, using the latest available sensitivity as a detached feature.This provides a landscape descriptor without propagating actor gradients through the critic feature.Source: D. Method 4: Augmented Critic (SG-CRT) · p. 9

VII. OPTIMIZATION INTERPRETATION OF SG-RL

SG-RL is interpreted as a composite descent update that combines stochastic PPO gradients with model-based solver guidance. The guidance can accelerate early optimization, but it is biased and need not improve immediate return during blending.

  • Maximizing expected return is equivalent here to minimizing expected cumulative performance loss, while PPO supplies a stochastic descent proxy.Source: §VII · p. 9
  • The lifted solver gradient is local model-based guidance from the surrogate loss, not the true policy gradient of expected closed-loop return.Source: A. Solver Guidance Signal and SG-RL… · p. 9
  • SG-RL combines PPO and solver-guidance signals through a composite update operator that can dynamically anneal solver influence.Source: A. Solver Guidance Signal and SG-RL… · p. 9
  • Solver gradients have lower variance but model bias, whereas sampled closed-loop PPO gradients reflect realized behavior but typically have higher early trajectory-level variance.Source: B. Early-Training Guidance · p. 10
  • Actor-side solver guidance is annealed so control progressively transfers back to PPO as learning advances.Source: B. Early-Training Guidance · p. 10
  • During blending, the composite update can differ from pure PPO and need not maximize immediate return improvement, although it may steer learning toward more promising regions.Source: C. Non-aligned Updates · p. 10

VIII. APPLICATION EXAMPLE: HIGH-SPEED AUTONOMOUS RACING

The paper evaluates weights-varying NMPC on two full-scale autonomous-racing platforms, where changing driving regimes, constraints, and nonlinear vehicle dynamics stress static cost tuning.

  • The validation uses Dallara AV-24 and Super Formula EAV-24 platforms modeled from real vehicle data and validated against physical racing cars.Source: §VIII · p. 10
  • Autonomous racing combines tight actuation and tire constraints, modeling uncertainty, nonlinearity, and strong regime shifts within a lap.Source: A. Why Autonomous Racing Stresses… · p. 10
  • A single static weight vector must compromise across straights, braking zones, corners, and corner exits, motivating online adaptation.Source: A. Why Autonomous Racing Stresses… · p. 10
  • The NMPC uses a nonlinear single-track model with Pacejka tire forces, aerodynamic effects, and explicit longitudinal and lateral vehicle dynamics.Source: B. Vehicle Dynamic Model for NMPC · p. 11
  • The controller tracks a time-optimal trajectory using stage and terminal quadratic costs over a reference segment supplied by an optimal-control planner.Source: C. NMPC Formulation · p. 11
  • The policy action directly parameterizes the diagonal NMPC weighting matrix, with weights fixed over each prediction horizon and updated between solves.Source: C. NMPC Formulation · p. 11
  • Strict actuator, state, powertrain, and combined-friction constraints enforce feasible vehicle operation.Source: C. NMPC Formulation · p. 11

D. Observations for Racing

The weight-selection policy combines current motion, recent tracking performance, and reference look-ahead, while bounded actions and solver sensitivities support closed-loop SG-RL updates.

IX. RESULTS

Closed-loop experiments under intentional model mismatch compare PPO, GB-PL, and SG-RL on two racing platforms. SG-RL improves return and sample efficiency without sacrificing physical closed-loop quality.

Experimental Setup

  • The experiments use high-fidelity plant and NMPC models identified from vehicle data but intentionally differing in fidelity, creating a deployment-relevant mismatch setting.Source: A. Experimental Setup · p. 12
  • The benchmark compares four SG-RL methods with PPO, a fixed-weight expert, reduced GB-PL, and the paper’s GB-PL baseline.Source: A. Experimental Setup · p. 12
  • PPO and SG-RL share the same RL setup, with PPO trained for 2M steps and SG-RL trained for 1M steps.Source: A. Experimental Setup · p. 12

Closed-Loop Performance

D. Generalization in Unseen Environments

SG-RL policies transfer zero-shot to unseen racelines and exhibit distinct context-dependent weight schedules. Ablations show that anticipation and performance history both support adaptation, while solver guidance remains useful with reduced temporal context.

Generalization in Unseen Environments

Weight Adaptation Dynamics

Ablations

H. Training-Dynamics Analysis

SG-RL accelerates PPO through selective, method-specific solver guidance rather than uniformly larger updates. Across two racing platforms, the methods improve closed-loop performance and sample efficiency, while their effectiveness depends on predictive-model fidelity and PPO-specific design choices.

Core PPO training dynamics

  • Winning SG methods outperform PPO through more structured updates, including better-timed and lower-magnitude steps.Table VIII compares PPO signals in early acceleration and later peak-performance windows.Source: H. Training-Dynamics Analysis · p. 16
  • In AV-24/Monza, SG-LOS improves early windowed return from −81.6 to −67.3, while SG-SCA raises peak mean return from −68.4 to −44.0.Both methods also reduce selected PPO update signals, including KL divergence and clip fraction.Source: H. Training-Dynamics Analysis · p. 16
  • In EAV-24/Yas Marina, SG-SCA wins the acceleration window by improving mean return from −111 to −93.6, while SG-LOS wins the peak window from −103 to −86.4.Both winning methods lower several PPO training losses and update-related quantities in their respective windows.Source: H. Training-Dynamics Analysis · p. 17

Method-specific guidance

  • Method-specific signals show selective solver guidance: SG-SCA brakes PPO steps on Monza but nudges PPO along solver-aligned directions on Yas Marina.SG-LOS retains nonzero guide loss in relevant windows, while SG-ADV and SG-CRT provide platform-specific advantage or critic reshaping.Source: H. Training-Dynamics Analysis · p. 17

Overall conclusions

  • SG-RL combines environment-based policy gradients with model-based solver sensitivities while preserving realized closed-loop return as the optimization objective.The framework uses solver guidance as an auxiliary signal for online MPC cost-weight adaptation.Source: §X · p. 17
  • Across two full-scale racing platforms, SG-RL matches PPO’s best return with up to 70.6% fewer samples, outperforms GB-PL by at least 54% in return, and generalizes zero-shot to unseen tracks.SG-SCA and SG-LOS achieve the strongest overall performance among the proposed variants.Source: §X · p. 17

Limitations and future work

  • SG-RL effectiveness depends on predictive-model fidelity, and the study does not systematically vary mismatch types or magnitudes; variants also add hyperparameters and use only on-policy PPO.Future work proposes physical-vehicle validation, off-policy extensions, and controlled model-mismatch studies.Source: §X · p. 17

Related papers

Loading 2609.01061v1…