OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real World

• OptLayer enforces arbitrary safety constraints on neural-network-predicted robot actions through a fully differentiable optimization layer.Source: §I · p. 1
• The method adds a simple reward strategy compatible with existing policy-optimization techniques.Source: §I · p. 1
• The approach is demonstrated on 3D reaching with collision avoidance for an industrial manipulator in simulation and the real world.Source: §I · p. 1
• Robot-control optimization can produce unsatisfactory policies when models are uncertain, constraints are overly conservative, or task dynamics are difficult to model.Source: §II · pp. 1–2
• Real-world reinforcement learning requires safe exploration during training and safe execution at inference time.Source: §II · p. 2
• Unconstrained reinforcement learning can involve unsafe actions and is highly sensitive to the reward and cost structure.Source: §I · p. 1, §III · p. 2

• An infinite-horizon discounted Markov Decision Process is defined by states, actions, transitions, rewards, initial-state probabilities, and a discount factor.Source: A. Reinforcement Learning Nomenclature · p. 2
• A trajectory consists of alternating states and actions, with actions sampled from the policy and subsequent states sampled from the transition distribution.Source: A. Reinforcement Learning Nomenclature · p. 2
• Qπ, Vπ, and Aπ denote the state-action value, value, and advantage functions, respectively.Source: A. Reinforcement Learning Nomenclature · p. 2
• TRPO iteratively refines a parameterized policy by solving a constrained optimization problem using sampled state-action trajectories.Source: A. Reinforcement Learning Nomenclature · p. 2
• The 2D reacher reward description includes target distance and a collision-penalty coefficient βcoll ∈{1, 5, 10, 50}.Source: Figure 2 · p. 3

Environment

• The 2-DoF robot must reach randomly sampled targets while avoiding a spherical obstacle in a planar environment.Episodes run for up to 200 steps with 0.01-second intervals, and actions specify joint-position increments.Source: B. Motivating Example · p. 3

Reward structure

• The environment rewards target proximity and penalizes collisions, with collision penalties parameterized by βcoll ∈ {1, 5, 10, 50}.The implementation monitors both external and self-collisions while neural-network policies interact with the environment.Source: B. Motivating Example · p. 3, C. Preliminary Results · p. 3

Results

• Reward evolution depends strongly on the collision-penalty weight.The comparison averages behavior across the tested penalty settings during training.Source: C. Preliminary Results · p. 3
• With βcoll = 1, collision avoidance is never learned because the penalty rcoll = −20 is outweighed by reaching and proximity incentives.The resulting strategy may hit the obstacle to restart the episode and obtain proximity reward.Source: C. Preliminary Results · p. 3
• With βcoll = 50, the penalty rcoll = −1000 reduces collisions but discourages exploration, leading the policy to remain near the start pose.The policy ignores further targets under this large penalty.Source: C. Preliminary Results · p. 3
• βcoll = 5 and 10 yield comparable rewards, but βcoll = 10 produces half the number of collisions.These results underscore the sensitivity of learned behavior to reward construction and the difficulty of learning safety without violations.Source: C. Preliminary Results · p. 3

Optimization formulation

• The quadratic program minimizes 1/2 x^T P x + q^T x subject to linear constraints.P and q define the objective, while G, h, A, and b define inequality and equality constraints.Source: A. OptLayer Architecture · p. 3

Pipeline

• The architecture is presented as a constrained optimization pipeline for enforcing safety constraints on predicted robot actions.The paper also describes the layer as fully differentiable and applicable to arbitrary action constraints.Source: Figure 4 · p. 4

Optimization formulation

• OptLayer seeks an action closest to the neural network’s prediction while satisfying linear equality and inequality constraints.The optimization uses a quadratic objective over action variables and constraints represented by matrix-vector systems.Source: A. OptLayer Architecture · pp. 3–4

Differentiability

• The objective and constraint matrices are differentiable expressions of the current state s_i and predicted action eai.This construction supports backward-pass gradients required for end-to-end neural-network training.Source: A. OptLayer Architecture · p. 4, B. Quadratic Program Initialization · p. 4

Constraint representation

• OptLayer accommodates robotic constraints that are constant, affine in state, auxiliary-variable dependent, or conditionally active.The exposition focuses on inequalities, while equality constraints are treated analogously.Source: B. Quadratic Program Initialization · p. 4

Objective matrices

• The optimization objective measures squared L2 distance between variables x and the predicted action eai.The objective keeps the corrected action close to the neural network’s output.Source: 1) Objective matrices · p. 4
• Because eai does not depend on x, its constant quadratic term does not affect the optimum.The objective can therefore be written in the standard quadratic-program form with P = I_nx and q = −eai.Source: 1) Objective matrices · p. 4

Constraint construction

• Joint-velocity limits become linear inequalities on the action hypothesis x = Δθi after approximating velocity by Δθi/ΔT.The resulting constraints use manufacturer-specified lower and upper velocity limits.Source: 2) Constant constraint matrices · p. 4
• For joint-velocity limits, the constant constraint matrices are Gcst = [I_nx, −I_nx]^T and hcst = [ΔT θ̇max, −ΔT θ̇min]^T.These matrices remain constant throughout the robot’s motion.Source: 2) Constant constraint matrices · p. 4

• Joint-position limits depend on the current joint positions because the next position is θ_i+1 = θ_i + x.Source: 3) Constraint matrices affine in the… · p. 4
• A selection matrix Hθ extracts joint-position terms from state s_i, making the corresponding bounds affine in the state.Source: 3) Constraint matrices affine in the… · p. 4
• Torque constraints use robot dynamics, with H(q) as the mass matrix and C(q, ˙q) as the bias vector of Coriolis, centrifugal, and gravitational forces.Source: 4) Auxiliary state parameters · p. 4
• The first six rows of the dynamics equation compute base forces, while rows indexed 7 and onward provide Hτ(q) and Cτ(q, ˙q) for joint torques.Source: 4) Auxiliary state parameters · p. 5
• Forward differences express velocity and acceleration through action x, making τ_i affine in x after combining the dynamics equations.Source: 4) Auxiliary state parameters · p. 5
• Because Hτ and Cτ are not simple functions of s_i, the environment computes them and appends their elements to the state for constraint construction.Source: 4) Auxiliary state parameters · p. 5

• Collision-avoidance constraints activate when a robot link’s distance d_l from the environment falls below the influence distance d_M.Source: 5) Conditional constraint matrices · p. 5
• The safety constraint limits how quickly d_l may decrease and requires d_l to increase strictly below the security distance d_m.Source: 5) Conditional constraint matrices · p. 5
• The state is extended with J(q,p_l)^T n_l and d_l so conditional constraint matrices and their activation test can be formed from s_i.Source: 5) Conditional constraint matrices · p. 5
• The activation condition depends only on state s_i, before the action x is optimized.Source: 5) Conditional constraint matrices · p. 5

Assembling constraint matrices for batch solving

• Always-active matrices are concatenated into G_base and h_base, while conditional matrices are appended only when their activation condition holds.Source: 1) Assembling constraint matrices for… · p. 5
• Fixed-size G and h matrices enable batch solving by substituting base submatrices when conditional constraints are inactive.Source: 1) Assembling constraint matrices for… · p. 5

Quadratic Program Solving

• The implementation solves quadratic programs with an interior-point method using objective and constraint matrices P, q, G, h, A, and b.Source: 2) Iterative resolution · p. 5

Iterative resolution

• BUILDTRAJ records states, predictions, rewards, values, corrected actions, and violation costs while OptLayer constrains each predicted action.Source: 2) Iterative resolution · p. 6, Figure 5 · p. 6

Quadratic Program Solving

• The solver iteratively updates primal, slack, and dual variables, using a constant k_max = 10 for batch solving because typical problems converge in about 5 iterations.Source: 2) Iterative resolution · pp. 5–6
• The final output is the action a* closest to the initial prediction ea that satisfies all safety constraints.Source: 2) Iterative resolution · p. 6

Trajectory Sampling

• OptLayer produces corrected actions a* for execution while retaining raw predictions ea to measure constraint violations.Source: A. Trajectory Sampling · p. 6
• Equality and inequality violation costs quantify how much each unconstrained prediction violates the corresponding constraints.Source: A. Trajectory Sampling · p. 6
• The total violation cost c = c_eq + c_in combines normalized equality and inequality contributions computed within OptLayer.Source: A. Trajectory Sampling · p. 6
• BUILDTRAJ outputs states, raw predictions, rewards, values, corrected actions, and constraint violation costs for constrained or unconstrained execution modes.Source: A. Trajectory Sampling · p. 6

Policy update strategies

• The four strategies are Unconstrained Predictions (UP), Constrained-learn Predictions (CP), Constrained-learn Corrections (CC), and Constrained-learn Predictions and Corrections (CPC).CP trains on raw predictions after constrained execution; CC trains on corrected actions; CPC trains on both raw predictions with discounted rewards and corrected actions with original rewards.Source: B. Policy Update Strategies · p. 6
• CPC first associates raw predictions with discounted rewards and then associates corrected actions with better rewards.This combines learning from the network’s original predictions with learning from the safe actions produced by OptLayer.Source: B. Policy Update Strategies · p. 6
• CP treats OptLayer as part of the environment from the network’s perspective by updating the policy with raw predicted actions.The network is trained using raw predictions rather than corrected actions.Source: B. Policy Update Strategies · p. 6
• CC trains directly on corrected actions, which can differ substantially from the neural network’s initial predictions.The paper attributes this distinction to the relationship between corrected actions and the stochastic policy being optimized.Source: B. Policy Update Strategies · p. 6

Experimental setting

• The experiments apply these strategies to 3D reaching with a 6-DoF industrial manipulator in both simulation and the real world.The setup uses an industrial manipulator equipped with a force-torque sensor and gripper; the task includes a target and obstacle.Source: §VI · p. 6, A. 3D Reaching with Obstacle Avoidance · p. 6, Figure 6 · p. 7

Simulation

• 700 average reward is reached in 5350 episodes by UP, 4950 by CP, and 3250 by CPC.The comparison comes from training over 7000 episodes on the simulated 3D reaching task.Source: B. Evaluating Policy Update Strategies · p. 7
• OptLayer ensures that collisions never happen for UP, CP, CC, and CPC during constrained training, whereas unconstrained training does not effectively learn collision avoidance.UP can learn to reach but still collides; CP achieves rewards comparable to UP while maintaining safety.Source: B. Evaluating Policy Update Strategies · p. 7, Figure 7 · p. 7
• CPC reaches performance comparable to UP and CP with fewer episodes than either strategy.The paper attributes CPC’s faster learning to first training on raw predictions with discounted rewards before training on corrected actions.Source: B. Evaluating Policy Update Strategies · p. 7

Real-world experiments

• On the real robot, CPC reaches a target while avoiding a balloon, unlike the unconstrained UP policy, which reaches the target but knocks the balloon down.The real-world setup also tests moving targets and moving obstacles.Source: C. Real-World Experiments · pp. 7–8
• A CPC policy trained for a specific task follows a moving target without hitting the obstacle and moves away when a moving obstacle approaches.These tests illustrate behavior under changing target and obstacle conditions without dedicated training for the moving-target task.Source: C. Real-World Experiments · p. 8, Figure 8 · p. 8
• The authors report that the system accommodates reasonable changes in environment and task conditions without dedicated training.The paper presents this as a result of the real-world robot experiments.Source: §VII · p. 8

Limitations

• Learning control policies from scratch may remain too time-consuming even for simple tasks.The authors identify this as a current implementation limitation and propose investigating robotics-specific action-space exploration.Source: §VII · p. 8