Pareto-Optimal Entropy-Regularized Trajectory Optimization

I. INTRODUCTION

• Robotic trajectory optimization seeks trajectories satisfying nonlinear dynamics, actuation limits, and safety constraints despite a highly non-convex landscape.Different local basins can represent qualitatively different motion strategies.Source: §I · p. 1
• DDP’s local second-order models are built around one nominal trajectory, making it vulnerable to suboptimal local minima and motivating exploration of alternatives.DDP exploits sequential dynamics structure and has linear complexity in the horizon.Source: §I · p. 1
• Sampling-augmented methods tie exploration to a small retained population, while scalar-cost ranking can discard candidates progressing on task performance or constraint satisfaction alone.These limits are especially consequential in constrained trajectory optimization.Source: §I · p. 2
• PER-DDP uses task-constraint Pareto selection to preserve alternatives while guaranteeing no selection-step degradation in augmented cost relative to DDP solutions.Its sampling policies use flexible priors derived from the free-energy/relative-entropy principle.Source: §I · p. 2
• PER-DDP reports higher success rates on bicycle, quadrotor, and Franka Panda systems, including environments no baseline solves.Its framework separates sampling modes, rollout evaluations, and retained candidates.Source: §I · p. 2

A. Notation

• The notation uses lowercase, bold lowercase, and bold uppercase for scalars, vectors, and matrices, respectively, and defines Gaussian and divergence notation.It also specifies derivative, norm, expectation, entropy, trace, determinant, and log-determinant conventions.Source: A. Notation · p. 2

B. Problem Statement

• The problem statement considers finite-horizon nonlinear discrete-time dynamics with a fixed initial state and state and control sequences constrained by the system dynamics.The nominal trajectory objective includes augmented costs for constraints, with hard feasibility handled separately.Source: B. Problem Statement · p. 2

C. Differential Dynamic Programming

• DDP locally decomposes the trajectory problem using Bellman recursion and a value function over the remaining horizon.Its Q-function relates the value at each time to a minimization over controls.Source: C. Differential Dynamic Programming · p. 2

V DDP

• DDP alternates a backward pass that computes local control gains with a forward dynamics rollout that updates the nominal trajectory until termination.The accepted rollout replaces the nominal trajectory.Source: C. Differential Dynamic Programming · p. 2, V DDP · pp. 2–3

D. Maximum-Entropy Differential Dynamic Programming

• ME-DDP replaces deterministic minimization with entropy-regularized policy optimization, whose optimal policy is a Gibbs distribution.The temperature λ sets the entropy scale.Source: D. Maximum-Entropy Differential… · p. 3
• Under a local Gaussian approximation, ME-DDP retains DDP’s mean and feedback gains while adding exploration through covariance λQ_uu,t^-1.The standard DDP backward pass is retained, while the forward pass samples around the current best-cost trajectory.Source: D. Maximum-Entropy Differential… · p. 3

E. Relaxed Log-Barrier Penalty Function

• The relaxed log-barrier remains finite and differentiable, making it suitable for infeasible samples while retaining DDP’s second-order nature.The classical log-barrier is singular at zero slack and undefined for negative slack.Source: E. Relaxed Log-Barrier Penalty Function · p. 3

F. Pareto Dominance and Pareto Front

• The Pareto front is the set of nondominated candidates, represented by objective vectors whose objectives are minimized.Pareto dominance compares candidates in the multi-objective space.Source: F. Pareto Dominance and Pareto Front · p. 3

A. Information-Theoretic Entropy Regularization

• The free-energy/relative-entropy formulation optimizes a control-sequence distribution to balance low trajectory cost against proximity to a reference density.The prior can encode directions suggested by the optimizer, dynamics, or environment, while λ controls the cost–prior tradeoff.Source: A. Information-Theoretic Entropy… · p. 3
• PER-DDP uses the optimizer structure to shape its reference law, yielding an optimization-aware sampling distribution q⋆.The variational principle is related to information-theoretic stochastic optimal control such as MPPI.Source: A. Information-Theoretic Entropy… · p. 3

B. Local DDP Form

• Bellman’s principle decomposes the complete-sequence variational objective into a stagewise update using the state-control function Qt.The trajectory cost is decomposed through Qt.Source: B. Local DDP Form · p. 3
• The local quadratic model is completed in δut to connect the stagewise variational update to DDP.The resulting local policy centers the quadratic cost; Quu,t determines exploration geometry, while pt shapes q⋆.Source: B. Local DDP Form · pp. 3–4

1) Bounded Uniform Baseline:

• A bounded-uniform prior over Ωt is a non-informative reference that drives qt toward a flat, maximally spread distribution within that region.The bounded region makes the uniform reference normalizable and confines q⋆ to Ωt.Source: 1) Bounded Uniform Baseline · p. 4
• Under the bounded-uniform prior, the posterior retains the ME-DDP Gaussian kernel with covariance λQ−1.The normalization is over Ωt.Source: 1) Bounded Uniform Baseline · p. 4
• The bounded-uniform form matches maximum-entropy DDP up to λ log |Ωt|, which leaves local value derivatives unchanged.This comparison is stated for the value propagation formulation.Source: 1) Bounded Uniform Baseline · p. 4
• Value propagation assumes weak truncation, with the optimal mean far from Ωt’s boundary relative to the Gaussian spread.Under this approximation, the mean retains the DDP value derivatives.Source: 1) Bounded Uniform Baseline · p. 4
• With weak truncation, the bounded-uniform baseline recovers ME-DDP using a proper least-informative prior and a Gaussian policy truncated to Ωt.The bounded support also satisfies the absolute-continuity condition for the KL formulation.Source: 1) Bounded Uniform Baseline · p. 4

2) Gaussian Baseline:

• The Gaussian baseline allows an informative affine prior control law and produces a Gaussian product policy.When the prior and DDP control laws are affine in δxt, the optimal mean is affine as well.Source: 2) Gaussian Baseline · p. 4
• A state-dependent prior feedback can modify Vxx through differences among the prior, posterior, and DDP gains.The derivation defines posterior-to-DDP and prior-to-posterior deviations before giving the value relations.Source: 2) Gaussian Baseline · p. 4
• Centering the prior mean at the DDP update removes mean-dependent corrections, while the KL term still regularizes policy covariance.The covariance may be annealed across optimization iterations.Source: 2) Gaussian Baseline · p. 4

3) Multimodal Gaussian Prior:

• A finite mixture of Gaussian priors represents multiple local expansions or basins.The prior assigns each mode a weight πr, equal to 1/R unless otherwise specified.Source: 3) Multimodal Gaussian Prior · p. 4
• The mixture structure is preserved in the local posteriors, with backward-pass approximations remaining specific to each mode.The sampling policy is over the local Gaussian posteriors.Source: 3) Multimodal Gaussian Prior · pp. 4–5
• The unnormalized mode masses are prior weights multiplied by local partition factors, and normalization yields the posterior mode weights.The partition-factor relation is connected to the free-energy formulation.Source: 3) Multimodal Gaussian Prior · p. 5
• Sampling selects mode r according to its posterior weight w(r).The mode-specific local derivatives remain associated with their respective state deviations.Source: 3) Multimodal Gaussian Prior · p. 5

1) Control Constraints:

• Box control bounds are handled directly rather than with cost penalties, which would distort Q_uu,t used in the sampling covariance.The bounds are Ω_t = {δu_t : u_min ≤ ū_t + δu_t ≤ u_max}.Source: 1) Control Constraints · p. 5
• Projection-based control updates define the local active set by solving the constrained problem at δx_t = 0, then reconstruct feedback gains from free and active sets.The resulting policy is affine for a fixed active set and piecewise affine in δx_t.Source: 1) Control Constraints · p. 5

2) State Constraints:

• State constraints use augmented costs with the relaxed log barrier, among several DDP constraint-handling approaches.The passage also lists penalties, augmented Lagrangian terms, projections, and barrier methods as alternatives.Source: 2) State Constraints · p. 5

E. Population-Based Policy Evaluation and Mode Selection

• PER-DDP decouples R sampling modes, M stochastic rollouts, and N reoptimization candidates, evaluating rollouts in objective space before selecting candidates.Existing methods typically use the same finite trajectory set as local solutions and sampling modes, with only a few rollouts promoted for reoptimization.Source: E. Population-Based Policy Evaluation… · p. 5

1) Sampling Modes R:

• Each sampling mode defines a local policy around its nominal trajectory, and R determines how many local trajectory regions are sampled.Source: 1) Sampling Modes R · p. 5

2) Rollout Samples M:

• The M rollout samples are distributed across the R modes to probe their local sampling policies.Source: 2) Rollout Samples M · p. 5

3) Pareto-Optimal Candidate Selection N:

• Pareto filtering preserves the minimum total-cost rollout, so its best total rollout cost is no worse than the deterministic DDP solutions.The zero-noise rollout of each mode is included among the evaluated samples.Source: 3) Pareto-Optimal Candidate Selection N · p. 5

F. Forward Pass

• In the forward pass, α scales both the feedforward step and perturbation, while K_q,t provides feedback stabilization.Source: F. Forward Pass · p. 5

• The algorithm takes the initial state and control sequence, horizon, iteration count, population sizes, regularization parameter, control constraints, and prior-barrier parameters as inputs.Source: §IV · p. 6
• After cold initialization, each iteration performs backward passes and mean rollouts for retained candidates, then selects sampling modes and generates stochastic rollouts.Source: §IV · p. 6
• Each iteration Pareto-prunes the existing front together with retained candidates and new rollouts, selects N candidates, and ultimately returns the final front and its minimum-cost member.Source: §IV · p. 6

1) Candidate Selection Operation:

• Candidate selection preserves anchors on the Pareto front or at key objective-space choices, then adds nearby fillers using control-space or covariance-weighted distance.The anchors include the minima of total cost and task cost, plus the normalized-front knee; the method prefers the knee to the potentially conservative constraint-cost minimum.Source: 1) Candidate Selection Operation · p. 6

2) Regularization:

• Q_uu,t is regularized before inversion using a nonnegative shift computed from its smallest eigenvalue.Source: 2) Regularization · p. 6

3) FD Capacity:

• The Pareto front is capped at F_max by iteratively removing trajectories least recently selected by candidate selection.Source: 3) FD Capacity · p. 6

4) Annealing:

• Each local mode uses a reduced-variance Gaussian prior with Hessian-aligned precision, which is exponentially annealed across DDP iterations.Source: 4) Annealing · p. 6

5) Sampling Frequency & Rejection Sampling:

• Sampling occurs every iteration by default, with configurable frequency; box-control sampling uses rejection with a fixed retry limit and clipping fallback.Source: 5) Sampling Frequency & Rejection… · p. 6

6) General Remarks:

• The general remarks allow alternative d and SELECTCANDIDATES choices in Algorithm 1 according to task requirements.Source: 6) General Remarks · p. 6

A. Experimental Setup

• The benchmarks cover constrained bicycle, quadrotor, and Franka Panda reaching tasks, with 297 environments each for BC and QD and 99 for FR.PER-DDP uses (N, M, Fmax) = (10, 100, 150) for BC/QD and (5, 50, 80) for FR.Source: A. Experimental Setup · pp. 6–7

B. Performance Evaluation and Benchmarking

• Pareto filtering preserves task–constraint tradeoffs that enable rollouts to uncover routes past obstacles, including candidates initially considered uncompetitive by total cost.The displayed baselines remain incomplete, consistent with exploration restricted by elite preservation and scalarized ranking.Source: B. Performance Evaluation and… · p. 7, Figure 3 · p. 8

VI. CONCLUSION

• PER-DDP shapes exploration with flexible control priors, preserves task–constraint alternatives through Pareto selection, and decouples rollout evaluation from local optimization.The study focuses on offline tasks in simulation; real-time MPC deployment and hardware validation are identified as next steps.Source: §VI · p. 7

B. Performance Evaluation and Benchmarking

• In representative BC and QD cases, proposed variants reach targets by distinct routes, while in FR they complete a safe reach where competing motions intersect an obstacle.The Fig. 2 caption reports that only proposed variants recover feasible solutions across all three environments.Source: Figure 2 · p. 7, B. Performance Evaluation and… · p. 7
• Across three benchmark systems, PER-DDP achieves higher success rates and finds solutions where all baselines fail.Its improved reliable success rates indicate that gains persist across repeated runs.Source: B. Performance Evaluation and… · p. 7, Table III · p. 8