PaperlayerView exact revision on arXiv

Source-linked AI summary

Accelerating Reinforcement Learning via MPC Solver-Gradient Guidance for Weights-varying MPC

Baha Zarrouki, Arslan Thobani, Jasper Hoffmann, Mattia Piccinini, Rudolf Reiter, Felix Jahncke, Sébastien Gros, Davide Scaramuzza, Johannes Betz

arXiv:2609.01061v1cs.ROcs.LGeess.SY

TL;DR

Fixed MPC weights can be suboptimal across changing conditions, while online adaptation is difficult because policy parameters influence long-horizon behavior through implicit constrained-MPC solutions. The paper introduces SG-RL, which adds bounded solver-gradient guidance to RL while retaining sampled closed-loop return as the objective; on full-scale racing platforms, it improves sample efficiency and return under model mismatch.

  • Problem

    Online MPC weight adaptation must handle changing performance trade-offs, but learning is difficult because constrained solver solutions create long-horizon, nonlinear dependencies on policy parameters.

  • Method

    SG-RL combines environment-based policy gradients with bounded solver-gradient guidance inside PPO while preserving realized closed-loop return as the optimization objective.

  • Results

    SG-RL matched PPO’s best return with up to 70.6% fewer training samples, outperformed GB-PL by at least 54% in return, and generalized zero-shot to unseen race tracks.

  • Takeaways & Limitations

    Environment-based policy gradients and model-based solver guidance are complementary signals for learning online MPC weight-adaptation policies.

  • Takeaways & Limitations

    GB-PL’s solver-derived gradient can systematically deviate from the true closed-loop improvement direction when the predictive model is inaccurate.

Abstract

from arXiv · show

In Model Predictive Control (MPC), cost-function weights shape closed-loop behavior, yet changing conditions often make fixed parametrizations suboptimal and motivate context-dependent online adaptation. Learning such policies is difficult because behavior depends implicitly on numerical MPC solutions, producing nonlinear, potentially nonsmooth, long-horizon dependencies on policy parameters. This creates a bias-variance tradeoff: Reinforcement Learning (RL) optimizes realized closed-loop return from environment samples but is sample-inefficient, whereas Gradient-Based Policy Learning (GB-PL) uses low-variance solver gradients from differentiable MPC to optimize surrogate losses on predicted trajectories but can be biased under model mismatch. We propose Solver-Gradient Guided Reinforcement Learning (SG-RL), a solver-sensitivity augmentation for RL-based online MPC cost-weight adaptation. SG-RL keeps sampled closed-loop return as the objective and uses bounded solver-derived gradients as auxiliary guidance to improve stability and sample efficiency. We instantiate SG-RL in Proximal Policy Optimization (PPO) with four modular algorithms that inject solver-gradient guidance into actor-update scaling, policy loss, advantage estimation, and value-function learning. On two full-scale autonomous racing platforms with intentional model mismatch, SG-RL reaches PPO's best closed-loop return with up to 70.6% fewer samples, outperforms GB-PL baselines by at least 54% in closed-loop return, and generalizes zero-shot to unseen environments.

I. INTRODUCTION

Fixed MPC cost weights can be inadequate across changing operating conditions, while learning online weight-adaptation policies is difficult because constrained MPC solutions create indirect, nonlinear, potentially nonsmooth, long-horizon dependencies. SG-RL combines sampled closed-loop policy gradients with bounded solver-gradient guidance, preserving return optimization while improving learning efficiency.

  • Motivation: Weights-varying MPC adapts cost weights online to preserve context-dependent trade-offs while keeping the underlying model and constraints unchanged.Different racetrack segments can require different performance trade-offs near handling limits.
  • Learning challenge: Learning online weight-adaptation policies is challenging because policy parameters affect returns through repeated constrained MPC solutions and closed-loop dynamics.This produces long-horizon credit assignment and implicit, nonlinear, potentially nonsmooth mappings.
  • Learning signals: Environment-based gradients reflect realized closed-loop objectives but can have high variance, whereas solver gradients offer lower-variance local guidance but inherit predictive-model bias.The complementary properties motivate combining both signals within one learning framework.
  • SG-RL: SG-RL uses solver-gradient guidance as an auxiliary signal for RL while continuing to optimize realized closed-loop return.This distinguishes it from approaches that optimize model-based surrogate objectives using solver sensitivities.
  • SG-RL: SG-RL is instantiated in PPO through four modular integrations: actor-update scaling, auxiliary policy learning, advantage estimation, and critic learning.The variants are evaluated separately to compare different forms of solver guidance.
  • Validation: All SG-RL variants outperform PPO in peak closed-loop performance, reach PPO’s best return with up to 70.6% fewer samples, and outperform GB-PL by at least 54% in return.Evaluation uses two full-scale autonomous-racing simulation platforms under intentional model mismatch and includes zero-shot generalization.

II. PROBLEM FORMULATION

The formulation uses a hierarchical weights-varying NMPC architecture: a high-level policy selects cost weights online, and a lower-level constrained NMPC solves and applies the first control input. Training optimizes long-horizon realized closed-loop performance, whose dependence on policy parameters creates a difficult outer optimization problem.

  • II. PROBLEM FORMULATION: Weights-varying NMPC places a high-level learning policy above a low-level receding-horizon controller that solves the constrained OCP at each sampling instant.The policy adapts cost weights along the task domain while the controller computes the applied input.
  • A. Weights-varying Nonlinear MPC: Online weight adaptation addresses changing operating regimes, disturbances, uncertainty, and task difficulty that make fixed tuning conservative or overly aggressive.Autonomous racing combines straights, braking zones, corners, and exits with different control priorities.
  • 2) Weights-varying NMPC OCP Formulation:: The NMPC OCP uses dynamics, inequality constraints, and stage and terminal costs parameterized by θ_t, while keeping the prediction model fixed.The weight vector is fixed over each prediction horizon but may change between sampling instants.
  • 3) Learning-based Cost-Weight Adaptation Policies:: A stochastic policy πϕ maps observation o_t to a distribution over NMPC weight vectors, with θ_t serving as the applied action.The policy runs during both training and deployment; only parameter updates distinguish the phases.
  • 3) Learning-based Cost-Weight Adaptation Policies:: At each time step, NMPC solves using the current state estimate and θ_t, then applies the first optimal control input in receding-horizon fashion.This couples high-level weight selection to the lower-level controller’s numerical solution.
  • B. Problem Statement: Learning Policies for Weights-varying NMPC: The upper-level objective is to optimize long-horizon closed-loop performance using instantaneous loss evaluated on the realized state and applied control.The performance loss may include collisions, failure-to-finish, or rapid-weight-variation penalties that differ from the internal NMPC cost.
  • B. Problem Statement: Learning Policies for Weights-varying NMPC: Policy learning is difficult because decisions affect future states, observations, solver calls, and returns indirectly through closed-loop interaction with the true plant.The resulting outer problem has long-horizon credit assignment around repeated constrained NMPC solves.
  • B. Problem Statement: Learning Policies for Weights-varying NMPC: The paper develops differentiable-optimization and reinforcement-learning baselines before introducing SG-RL to combine their strengths.The approaches target the upper-level optimization problem for the weight-selection policy.

III. DESIGNING A NEURAL POLICY FOR WEIGHTS-VARYING NMPC

The neural policy maps history-aware observations to bounded, context-dependent NMPC cost weights, allowing reactive, anticipatory, and mismatch-adaptive behavior while preserving valid cost parameters.

  • Policy architecture: The policy maps high-dimensional observations to cost weights for the next NMPC planning horizon, enabling online adaptation when task requirements and disturbances change.The hierarchical design leaves constraints and low-level dynamics to NMPC while the neural policy shapes broader behavioral objectives.
  • History-aware inputs: The observation combines current measurable context, reference preview, and recent performance history to support reactive, proactive, and adaptive weight changes.Reference preview anticipates braking, curvature, or acceleration events, while performance history helps infer persistent disturbances, model mismatch, or actuator transients.
  • Constrained outputs: Policy outputs are restricted to strictly positive, bounded ranges to preserve conditioning, positive-definiteness for diagonal costs, and numerical stability.Unconstrained logits are mapped into domain-specified limits using a differentiable scaled-tanh parameterization.
  • Constrained outputs: The bounded mapping regularizes the search space and avoids hard clipping, but attenuates gradients near parameter boundaries.The constraints confine optimization to a bounded hyper-rectangle and remove irrelevant parameter regions.
  • Architecture comparison: The proposed architecture uniquely combines reactivity, adaptability, and proactive anticipation within a constrained action space.This comparison is stated in the policy-architecture table caption.

B. Gradient Propagation and Policy Learning

GB-PL differentiates a smooth surrogate loss through the NMPC solver and policy, while PPO learns from realized closed-loop returns; their trade-offs motivate combining solver guidance with reinforcement learning.

  • Gradient propagation: GB-PL backpropagates a differentiable surrogate loss evaluated on the NMPC-predicted trajectory through solver sensitivities and the neural policy.The chain rule combines loss gradients, solver sensitivity, and the policy Jacobian to obtain a weight-space solver gradient and update policy parameters.
  • GB-PL limitations: GB-PL is restricted to objectives with smooth trajectory-based surrogates, excluding or approximating genuinely nonsmooth and sparse event-based terms.Examples include “Lap completed” and “Collision” objectives.
  • GB-PL limitations: GB-PL performs local descent without explicit exploration, making training sensitive to initialization and the surrogate landscape.This limits its ability to address the myopic nature of local gradient descent.
  • GB-PL limitations: GB-PL derives sensitivities from open-loop model predictions, so inaccurate predictive models can make its gradient deviate from the true closed-loop improvement direction.The method primarily tunes MPC costs to match task loss under the solver’s predictive model rather than directly optimizing realized plant behavior.
  • RL baseline: PPO treats the plant and NMPC controller as the closed-loop environment and updates its stochastic weight policy from sampled returns.The reward is evaluated on realized closed-loop states and applied inputs, not the internal NMPC cost or predicted trajectory.
  • Learning-signal trade-off: PPO’s environment-based signal reflects exploration, stochasticity, and model mismatch, complementing the lower-variance but model-dependent solver gradients.The comparison uses aligned task-level rewards so differences reflect learning signals and update mechanisms rather than different objectives.

VI. SOLVER-GRADIENT GUIDED REINFORCEMENT LEARNING

SG-RL injects bounded NMPC solver-gradient guidance into PPO while retaining sampled closed-loop learning. Four variants place this guidance in distinct PPO components to compare their effects.

  • SG-RL injects the model-based NMPC solver loss gradient gSG,t into selected components of PPO.
  • The solver signal is normalized so ∥gSG,t∥2 ≤1, making its direction comparable across contexts while method-specific coefficients control update magnitude.When sensitivities are unavailable, SG-RL sets gSG,t = 0 and recovers standard PPO behavior.
  • Four hybrid methods inject solver guidance through actor-update scaling, performance-gated policy loss, advantage shaping, and critic augmentation.The variants isolate different PPO injection points: SG-SCA, SG-LOS, SG-ADV, and SG-CRT.
  • Adaptive PPO Update Scaling (SG-SCA): SG-SCA scales the PPO update magnitude according to alignment between solver and PPO loss gradients without changing the PPO update direction.Its scaling factor is bounded to prevent excessive steps and cannot reverse the PPO direction.

B. Method 2: Performance-Gated Loss (SG-LOS)

SG-LOS selectively supplements PPO with a solver-guided correction when sampled transitions underperform, while avoiding uniform reliance on a local model-based surrogate.

  • SG-LOS adds an explicit target-following loss to PPO when the RL agent performs poorly.Unlike SG-SCA, it adds a loss term rather than rescaling the PPO descent step.
  • The method avoids globally regularizing PPO with a local, open-loop, model-based surrogate whose reliability may vary across samples.Solver guidance is instead used to construct a local correction target for selected transitions.
  • SG-LOS constructs a target weight configuration by taking a fixed step ηguide along the normalized solver descent direction −gSG,t.
  • The current policy mean is penalized for deviating from the solver-suggested target through a gated guide loss.
  • Negative-advantage samples receive corrective solver guidance, whereas positive-advantage samples reduce to standard PPO.Transitions with negligible solver-gradient norm are masked out, and the gate activates more often early in training when underperforming transitions are common.

C. Method 3: Advantage Shaping (SG-ADV)

SG-ADV incorporates solver-gradient consistency into PPO’s advantage estimates, rewarding policy-mean displacements aligned with local solver descent while leaving the critic architecture unchanged.

  • SG-ADV injects solver guidance into the advantage estimate while keeping PPO on-policy.It targets exploration rather than directly modifying the optimizer step.
  • The shaped advantage adds a correction based on alignment between the policy’s mean weight displacement and the local solver loss landscape.
  • A positive projection onto −gSG,t indicates that the policy mean moves toward lower surrogate performance loss.
  • SG-ADV rewards solver-consistent action displacements and penalizes solver-inconsistent ones through the advantage signal.
  • Augmented Critic (SG-CRT): SG-CRT instead augments the critic with a residual correction from a normalized solver-gradient encoder, using the latest available sensitivity as a detached feature.This provides a landscape descriptor without propagating actor gradients through the critic feature.

VII. OPTIMIZATION INTERPRETATION OF SG-RL

SG-RL is interpreted as a composite descent update that combines stochastic PPO gradients with model-based solver guidance. The guidance can accelerate early optimization, but it is biased and need not improve immediate return during blending.

  • Maximizing expected return is equivalent here to minimizing expected cumulative performance loss, while PPO supplies a stochastic descent proxy.
  • The lifted solver gradient is local model-based guidance from the surrogate loss, not the true policy gradient of expected closed-loop return.
  • SG-RL combines PPO and solver-guidance signals through a composite update operator that can dynamically anneal solver influence.
  • Solver gradients have lower variance but model bias, whereas sampled closed-loop PPO gradients reflect realized behavior but typically have higher early trajectory-level variance.
  • Actor-side solver guidance is annealed so control progressively transfers back to PPO as learning advances.
  • During blending, the composite update can differ from pure PPO and need not maximize immediate return improvement, although it may steer learning toward more promising regions.

VIII. APPLICATION EXAMPLE: HIGH-SPEED AUTONOMOUS RACING

The paper evaluates weights-varying NMPC on two full-scale autonomous-racing platforms, where changing driving regimes, constraints, and nonlinear vehicle dynamics stress static cost tuning.

  • The validation uses Dallara AV-24 and Super Formula EAV-24 platforms modeled from real vehicle data and validated against physical racing cars.
  • Autonomous racing combines tight actuation and tire constraints, modeling uncertainty, nonlinearity, and strong regime shifts within a lap.
  • A single static weight vector must compromise across straights, braking zones, corners, and corner exits, motivating online adaptation.
  • The NMPC uses a nonlinear single-track model with Pacejka tire forces, aerodynamic effects, and explicit longitudinal and lateral vehicle dynamics.
  • The controller tracks a time-optimal trajectory using stage and terminal quadratic costs over a reference segment supplied by an optimal-control planner.
  • The policy action directly parameterizes the diagonal NMPC weighting matrix, with weights fixed over each prediction horizon and updated between solves.
  • Strict actuator, state, powertrain, and combined-friction constraints enforce feasible vehicle operation.

D. Observations for Racing

The weight-selection policy combines current motion, recent tracking performance, and reference look-ahead, while bounded actions and solver sensitivities support closed-loop SG-RL updates.

  • The policy observes current motion through stacked recent longitudinal velocities and yaw-rate measurements to capture dynamic regimes and short transients.
  • Performance history supplies recent path-tracking deviations, allowing the policy to use temporal execution information.
  • Task anticipation supplies uniformly spaced reference-velocity samples over the NMPC horizon as look-ahead features.
  • The actor outputs bounded cost weights that parameterize the NMPC weighting matrices, while the differentiable solver provides weight sensitivities for SG-RL guidance.
  • The fixed performance loss evaluates realized velocity and lateral errors plus jerk and steering-rate penalties, and the reward is its negative.

IX. RESULTS

Closed-loop experiments under intentional model mismatch compare PPO, GB-PL, and SG-RL on two racing platforms. SG-RL improves return and sample efficiency without sacrificing physical closed-loop quality.

  • Experimental Setup: The experiments use high-fidelity plant and NMPC models identified from vehicle data but intentionally differing in fidelity, creating a deployment-relevant mismatch setting.
  • Experimental Setup: The benchmark compares four SG-RL methods with PPO, a fixed-weight expert, reduced GB-PL, and the paper’s GB-PL baseline.
  • Experimental Setup: PPO and SG-RL share the same RL setup, with PPO trained for 2M steps and SG-RL trained for 1M steps.
  • Closed-Loop Performance: SG-SCA improved PPO’s peak mean reward by 33.2%, while SG-LOS, SG-CRT, and SG-ADV improved it by 20.4%, 18.5%, and 14.1%, respectively.
  • Closed-Loop Performance: 70.6% fewer samples were needed by SG-SCA and SG-LOS to surpass PPO’s best mean return on EAV-24/Yas Marina.
  • Closed-Loop Performance: SG-RL best-return gains over the paper’s GB-PL baseline were 54.3%–59.5%, while gains over reduced GB-PL were 68.5%–72.0%.
  • Closed-Loop Performance: PPO and SG-SCA achieved the tightest tracking and lowest maximum steering rates, indicating that SG-RL’s sample-efficiency gains did not sacrifice closed-loop quality.

D. Generalization in Unseen Environments

SG-RL policies transfer zero-shot to unseen racelines and exhibit distinct context-dependent weight schedules. Ablations show that anticipation and performance history both support adaptation, while solver guidance remains useful with reduced temporal context.

  • Generalization in Unseen Environments: Policies trained on one circuit receive local state and short-horizon reference information rather than track coordinates, enabling zero-shot evaluation on unseen racelines.
  • Generalization in Unseen Environments: On unseen tracks, SG-RL improves over GB-PL by 65.6%–68.6% on Laguna Seca and 56.1%–62.1% on Suzuka.
  • Generalization in Unseen Environments: The transfer results indicate a learned map from local driving context to NMPC weight rebalancing rather than raceline overfitting.
  • Weight Adaptation Dynamics: Correlation analyses identify associations between grouped observations and weights, while temporal schedules reveal method-specific weight co-variation along the lap.
  • Weight Adaptation Dynamics: PPO uses coarse event-wise scheduling, whereas SG-SCA applies sparse localized activations near curvature and deceleration zones.
  • Weight Adaptation Dynamics: SG-LOS co-schedules acceleration, jerk, and steering-rate objectives while keeping lateral and velocity weights small except for short corrections.
  • Weight Adaptation Dynamics: GB-PL most strongly varies velocity and steering-smoothness weights, with its velocity-weight trajectory nearly reproducing the reference-velocity profile.
  • Ablations: Removing anticipation makes policies feedback-only and prevents anticipation of curvature or deceleration, systematically degrading performance.

H. Training-Dynamics Analysis

SG-RL accelerates PPO through selective, method-specific solver guidance rather than uniformly larger updates. Across two racing platforms, the methods improve closed-loop performance and sample efficiency, while their effectiveness depends on predictive-model fidelity and PPO-specific design choices.

  • Core PPO training dynamics: Winning SG methods outperform PPO through more structured updates, including better-timed and lower-magnitude steps.Table VIII compares PPO signals in early acceleration and later peak-performance windows.
  • Core PPO training dynamics: In AV-24/Monza, SG-LOS improves early windowed return from −81.6 to −67.3, while SG-SCA raises peak mean return from −68.4 to −44.0.Both methods also reduce selected PPO update signals, including KL divergence and clip fraction.
  • Core PPO training dynamics: In EAV-24/Yas Marina, SG-SCA wins the acceleration window by improving mean return from −111 to −93.6, while SG-LOS wins the peak window from −103 to −86.4.Both winning methods lower several PPO training losses and update-related quantities in their respective windows.
  • Method-specific guidance: Method-specific signals show selective solver guidance: SG-SCA brakes PPO steps on Monza but nudges PPO along solver-aligned directions on Yas Marina.SG-LOS retains nonzero guide loss in relevant windows, while SG-ADV and SG-CRT provide platform-specific advantage or critic reshaping.
  • Overall conclusions: SG-RL combines environment-based policy gradients with model-based solver sensitivities while preserving realized closed-loop return as the optimization objective.The framework uses solver guidance as an auxiliary signal for online MPC cost-weight adaptation.
  • Overall conclusions: Across two full-scale racing platforms, SG-RL matches PPO’s best return with up to 70.6% fewer samples, outperforms GB-PL by at least 54% in return, and generalizes zero-shot to unseen tracks.SG-SCA and SG-LOS achieve the strongest overall performance among the proposed variants.
  • Limitations and future work: SG-RL effectiveness depends on predictive-model fidelity, and the study does not systematically vary mismatch types or magnitudes; variants also add hyperparameters and use only on-policy PPO.Future work proposes physical-vehicle validation, off-policy extensions, and controlled model-mismatch studies.
Loading 2609.01061v1…