PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimal Control with Learned Critics under Unmodeled State Dependencies

Philipp Schoch, Markus Ryll

arXiv:2610.05359v1cs.ROcs.AI

TL;DR

MPC is constrained by the need for tractable analytical dynamics and finite horizons, while model-free RL is typically sample-inefficient. This paper embeds residual dynamics learning and a learned critic in a GPU-batched iLQR planner. The approach improves reported control performance, including a higher success rate than nominal MPC on payload quadrotor hovering.

  • Problem

    MPC requires optimization-friendly analytical dynamics and can be myopic, while model-free RL avoids explicit models but is sample-inefficient.

    Source: §1 · pp. 1–2

  • Method

    The framework augments nominal dynamics with a learned residual and blends a learned critic into constrained MPC using a GPU-batched iLQR solver.

    Source: §1 · p. 2, §3.1 · p. 3, §3.3 · p. 4

  • Results

    The learned-critic quadrotor variant reached 0.79 success rate versus 0.05 for nominal MPC under an unmodeled suspended payload.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The learned critic was the main driver of observed gains, supplying corrective descent directions and long-horizon information unavailable to finite-horizon MPC alone.

    Source: §7 · p. 9

    Optimization is sensitive to irregular learned-model derivatives, and learned critics outperformed nominal baselines only with modest λ reductions, roughly 0.98 to 0.9, in the reported experiments.

    Source: §6 · p. 9

Abstract

from arXiv · show

Model Predictive Control (MPC) provides a structured and constraint-aware mechanism for decision-making, but its reliance on optimization-friendly analytical dynamics models limits its use in tasks with contacts and other hard-to-model state dependencies. Model-free reinforcement learning avoids explicit modeling assumptions but typically requires large amounts of interaction data. We present a learning-based MPC framework that combines the data efficiency and structure of local model-based planning with learned components that compensate for incomplete dynamics and finite-horizon myopia. The method augments a nominal analytical model with a residual dynamics network that learns missing state-dependent effects from data and combines the resulting planner with a learned action-value critic that injects long-horizon MDP structure into the local iLQR optimization. To make this practical at reinforcement-learning scale, we develop a GPU-accelerated batched iLQR solver that evaluates learned dynamics and critic networks inside the optimal-control loop and solves thousands of trajectory-optimization problems in parallel. The complete system is integrated into a robotics simulator, enabling scalable model-based reinforcement learning under incomplete dynamics. Experiments on biased and incompletely modeled control tasks show that the approach improves closed-loop control performance while preserving the model-based structure needed for efficient constrained trajectory optimization.

1 Introduction

The paper addresses MPC’s dependence on tractable analytical dynamics and finite horizons, alongside model-free RL’s sample inefficiency. It combines residual dynamics learning and a learned critic within a GPU-batched iLQR planner, then develops the blending, training, and feature-guidance components that support this approach.

1 Introduction

  • MPC depends on tractable dynamics and finite horizons, while model-free RL can use high-dimensional observations but is sample-inefficient.Contact dynamics and short horizons can undermine MPC; model-free methods also struggle with high-dimensional states and sparse rewards.Source: §1 · pp. 1–2

2 Preliminaries

  • Classical MPC’s terminal-cost approximation can miss global MDP structure, especially when a task requires reaching an intermediate subgoal.The paper frames planning as a finite-horizon approximation to an infinite-horizon MDP.Source: §1 · p. 1, §2 · p. 2

3.1 Blending the Learned Critic

  • The blended Q-estimate combines a learned global critic with the nominal MPC rollout, trading model bias against critic approximation bias.At λ = 0 it recovers the critic; at λ = 1 it recovers the standard MPC rollout, with intermediate values retaining local model accuracy.Source: §3.1 · p. 3

3.2 Learning the Critic

  • Critic learning uses Bellman-error training with rollout or MPC continuation targets, balancing stale offline values against expensive fresh solves.An ensemble supplies target values, while disagreement between value heads and their gradients estimates uncertainty and attenuates critic influence at inference.Source: §3.2 · pp. 3–4

3.3 Learning the Dynamics Residual

  • The residual model learns missing dynamics for directly observable state components, and rollout-consistency loss penalizes error accumulated during integration.Training alternates between collecting rollouts, fitting residual dynamics, and updating the critic.Source: §3.3 · p. 4

3.4 Guiding the Critic

  • Optional task-specific context and features let learned modules operate in task coordinates rather than embodiment-specific raw states.For manipulation, features may include object-to-end-effector position, goal-to-object position, and end-effector velocity.Source: §3.4 · pp. 4–5

4 OCP Solver

The solver uses DDP-style iLQR to optimize a rolled-out trajectory, wraps the primal solve in an Augmented Lagrangian Method for constraints, and blends learned-critic values into the recursion.

4.1 iLQR

  • iLQR rolls out a reference trajectory, builds local quadratic action-value models in a backward pass, and uses line search to accept control updates.The implementation batch-computes derivatives and uses feedback gains in candidate rollouts.Source: §4.1 · p. 5

4.2 Augmented Lagrangian Method

  • The Augmented Lagrangian Method alternates an iLQR primal solve with dual updates, increasing penalties for constraints that remain violated.Inactive inequalities contribute no residuals, gradients, or Hessians; when multiplier updates vanish, residuals approach zero and stationarity reduces to the original problem’s KKT conditions.Source: §4.2 · p. 6

4.3 Blended-iLQR

  • Blended-iLQR substitutes the ALM stage objective into the critic-blended recursion, combining constraint terms and critic derivatives while retaining iLQR’s dynamic-programming form.The next-state value contribution is scaled by λγ, and the continuation is omitted at the last control step.Source: §4.3 · p. 6

5 Results

Experiments test solver throughput, critic blending, and control with a biased quadrotor model and incompletely modeled cube-stacking contacts. The learned-critic planner improves quadrotor and manipulation performance over their nominal MPC baselines.

Speed Benchmarking

  • Batched GPU execution was roughly one to two orders of magnitude faster than CPU execution over the relevant batch-size range when learned dynamics and critic models were included.Amortized runtime per problem fell as batch size increased to about 10^4 parallel QPs before saturating; automatic differentiation remained close to analytical derivatives.Source: §5 · p. 6

Isolating the Effect of the Critic Blend

  • Lowering λ_f toward zero recovered optimal-policy success rate and cost at all tested horizons, while all-stage critic blending reached that performance with λ closer to 1.Without the critic, MPC failed for horizons H ∈ {2, 5, 10} and succeeded at H = 20; Figure 2 plots success rate and mean episode cost for both blend settings.Source: Figure 2 · p. 7, §5 · p. 7

Overcoming Biased Dynamics and Unmodeled State-Dependencies

  • Success rate rose from 0.05 to 0.79 and mean cost fell from 2.6e4 to 4.7e3 with the learned-critic quadrotor variant, versus nominal MPC.The idealized INDI baseline reached 0.94 success rate and 5.6e3 mean cost; Figure 3 shows success increasing and cost dropping as λ decreases.Source: Figure 3 · p. 8, §5 · p. 8

Learning to Plan over Contacts Using Visual and Haptic Information

  • The cube-stacking success rate rose from 0.361 for the informed nominal baseline to 0.401 with residual grasp-score learning and 0.94 with a learned critic.The critic is conditioned on visual depth and haptic observations; reported recovery behaviors include correcting failed grasps and poor approaches.Source: §5 · pp. 8–9

6 Limitations

The solver’s dependence on learned-model derivatives and sequential horizon computations constrains optimization reliability and single-system replanning speed.

6 Limitations

  • Optimization is sensitive to irregular first- and second-order derivatives of the learned residual and critic models.With learned critics, only modest reductions from λ = 1, roughly 0.98 to 0.9, outperformed nominal baselines in the reported experiments.Source: §6 · p. 9
  • Poor gradients can steer optimization into uninformative state-space regions, degrade early data collection, and stall training.The authors identify stronger training-time regularization, efficient off-policy target construction, and replay-buffer priority sampling as directions to investigate.Source: §6 · p. 9
  • Sequential rollout, Riccati recursion, and line search preclude replanning above 100 Hz on a single system in the current implementation.The time-indexed local feedback policy can still close the loop at the control rate while the planner runs more slowly.Source: §6 · p. 9

7 Conclusion

The conclusion attributes the observed gains chiefly to integrating a learned critic into gradient-based MPC, where it supplies long-horizon behavioral information. The appendix interprets the blended objective as the critic plus weighted Bellman-residual corrections along the planned trajectory.

7 Conclusion

  • Across runtime and control experiments, the framework scaled to thousands of parallel optimal-control problems and improved performance under biased and incomplete dynamics.The conclusion identifies stronger critic pre-training, including representation learning from video demonstrations, as future work.Source: §7 · p. 9

A Bellman-Residual Form of the Blended Objective

  • The blended objective equals the learned critic plus geometrically weighted local Bellman-residual corrections along the planned trajectory.When the critic satisfies the Bellman relation along the rollout, residuals are small; disagreement between the local model-based rollout and critic accumulates corrections.Source: A Bellman-Residual Form of the Blended… · p. 10

B Quadratic Q Expansion for blended ALM-iLQR

The ALM-iLQR quadratic Q expansion incorporates learned Q-function terms into its derivatives and carries value-function derivatives through the recursion.

C Runtime Scaling Benchmark

The benchmark evaluates batched iLQR runtime with learned models and compares GPU and CPU execution. Its results show a large GPU advantage for learned-model solves, while a point-mass experiment sweeps critic-blend settings.

C Runtime Scaling Benchmark

  • The benchmark measures one iLQR iteration on 8-dimensional states, 4-dimensional actions, and a 32-step horizon, averaging 20 runs after two warmups.Runs use a mobile GPU and a Ryzen 9 CPU.Source: C Runtime Scaling Benchmark · p. 11
  • Learned-model solves run roughly one to two orders of magnitude faster on GPU than CPU, with per-QP runtime gains flattening near 104 parallel problems.The saturation coincides with the practical hardware limit of the tested mobile GPU.Source: Figure 4 · p. 11, C Runtime Scaling Benchmark · pp. 11–12

D Point-Mass Experiment Details

  • The point-mass experiment uses a 2 kg planar system and sweeps all-stage and terminal-only critic-blend weights across parallel environments.Each blend value is held for one episode; the all-stage sweep uses 10^3 environments.Source: D Point-Mass Experiment Details · p. 12

E.1 Task Setup

The quadrotor task combines a biased nominal rigid-body model with suspended-load dynamics omitted from that model, while constraining virtual controls and using a hover-tracking cost.

  • Virtual controls have componentwise box constraints, and the quaternion is optimized in a three-dimensional tangent coordinate.Source: §E.1 · p. 12
  • The simulator adds observed swing-load coordinates for a 0.172 kg suspended load, but omits their effects from the nominal rigid-body dynamics.The residual effects on translation and rotation must be learned from data.Source: §E.1 · pp. 12–13
  • The nominal model has mass and inertia bias and does not represent the load-related state dependencies in its rigid-body dynamics.Source: §E.1 · p. 13
  • The task uses a quadratic hover-tracking stage cost and compares MPC variants with an idealized INDI baseline using noise- and delay-free acceleration measurements.Source: §E.1 · p. 13

E.2 Experimental Insights

Under biased and incomplete quadrotor dynamics, the learned critic controller avoids the nominal MPC failure observed in representative and randomized rollouts. The section also examines local critic behavior and the cube-stacking task setup.

E.2 Experimental Insights

  • The learned critic avoids nominal MPC's late-flight tip-over by favoring a lower-velocity approach that keeps suspended-load motion bounded.Nominal MPC initially reduces position error but excites the load and crashes late in the episode.Source: §E.2 · p. 13, Figure 5 · p. 14
  • Across 300 randomized rollouts, learned-controller trajectories converge toward hover as dispersion contracts, while nominal MPC develops late dispersion and downward z drift.Source: §E.2 · p. 13, Figure 6 · p. 15

F.1 Task Setup

  • The cube-stacking task asks the robot to pick up a target cube and place it on a support cube, with target transport scaled by a learned grasp gate.Near zero, the target is effectively decoupled from the end effector; near one, it follows commanded motion.Source: §F.1 · p. 15

E.2 Experimental Insights

  • The four value heads agree in the plotted x-y slice, indicating low ensemble uncertainty for the critic correction at that rollout state.Only planar position varies in the slice; the other state and action entries are fixed.Source: §E.2 · p. 14, Figure 7 · p. 16
  • At fixed velocity, attitude, and payload state, the critic shifts the local optimum relative to stage cost while remaining smooth and avoiding distant low-value basins.Value-head variance sets the scale for blending the critic with the nominal MPC objective.Source: §E.2 · p. 14, Figure 8 · p. 16

F.1 Task Setup

  • Contact dynamics are not modeled explicitly in the manipulation task, so their effects must be absorbed by the learned residual and value function.The residual model uses state, control, contact forces, grasp score, and optional image features.Source: §F.1 · pp. 16–17
  • The table captions report that the learned critic maintains strong cube-stacking performance at short horizons, while depth and haptic inputs improve success rate monotonically.Source: Table 1 · p. 17, Table 2 · p. 17

F.2 Experimental Insights

The experiments find that critic learning with informative observations drives the main gains, while residual dynamics learning helps only moderately on cube stacking. Critic learning improves performance across planning horizons, and depth and haptic inputs strengthen the critic's results.

Experimental insights

  • The ablations identify critic learning with informative observations as the main benefit, while residual dynamics learning provides only moderate improvement on this task.This is the overall interpretation of Tables 1, 2, and 3.Source: §F.2 · p. 17

Effect of planning horizon

  • 0.685 success rate at H = 5 contrasts with nominal MPC, which barely succeeds there; the learned critic improves success across the tested horizon range.Nominal MPC works best with a medium-length horizon; the learned critic also degrades at long horizons as imperfect gradients and curvature can destabilize the backward pass.Source: §F.2 · p. 17

Effect of visual and haptic inputs

  • 0.941 success rate is reported with state, action, goal, depth, and haptic inputs, versus 0.543 with depth and 0.23 with only state, action, and goal features.The authors attribute the consistent gain to reduced ambiguity about grasp quality and task phase; Figure 9 illustrates the transport phase and predicted cube and end-effector trajectories.Source: §F.2 · p. 17, Figure 9 · p. 18

Effect of residual dynamics learning

  • Residual dynamics variants do not improve significantly over the nominal baseline and often increase cost on cube stacking.The authors suggest the phased nominal model is sufficient once task phase is inferred, while added residual dynamics can introduce non-convexity and erratic motions that risk dropping the cube.Source: §F.2 · pp. 17–18, Table 3 · p. 18

G Solver and Learning Settings

The appendices summarize shared solver and learning settings, task-specific quadrotor and cube-stacking configurations, and PPO hyperparameters.

Solver and learning settings

Shared settings

  • Table 4 lists shared solver, optimization, and data-collection settings used in both tasks.Source: Table 4 · p. 19

Task-specific and PPO settings

H Quaternion Projection

The solver supports states containing both Euclidean variables and quaternions by handling rotations in local coordinates. It renormalizes quaternions during rollout and projects derivatives to local tangent coordinates in the backward pass.

Quaternion handling

  • Quaternion states are renormalized after numerical integration, and line-search displacements use Euclidean differences for vector blocks and local rotation errors for quaternion blocks.Source: H Quaternion Projection · p. 19

Derivative projection

  • Direct quaternion-coordinate perturbations introduce radial components that violate ∥q∥ = 1, so the backward pass projects derivatives into local tangent coordinates.Source: H Quaternion Projection · p. 20

Geometric Jacobian

Local rotation expansion

  • The rotational expansion is constructed around a unit quaternion using a local Rodrigues perturbation and the Cayley parametrization.Source: H Quaternion Projection · p. 20

Hessian projection

  • The Hessian projection reuses the first-order projection matrix and omits second-order curvature terms, following the iLQR Gauss-Newton approximation.Source: H Quaternion Projection · p. 20

Related papers

Loading 2610.05359v1…