PaperlayerView exact revision on arXiv

Source-linked AI summary

Conflict-Averse Gradient Aggregation for Constrained Multi-Objective Reinforcement Learning

Dohyeong Kim, Mineui Hong, Jeongho Park, Songhwai Oh

arXiv:2403.00282v2cs.LG

TL;DR

Deep-RL linear scalarization can converge to local optima, while managing safety constraints can make training unstable. CoMOGA transforms objectives into constraints and incorporates safety constraints in a linearized policy update. The paper reports CP-optimal convergence in tabular settings and strong experimental performance with constraint satisfaction across tasks.

  • Problem

    With deep function approximators, linear scalarization tends to converge to local optima, while concurrent policy and multiplier updates for constraints can destabilize training.

    Source: §1 · pp. 1–2

  • Method

    CoMOGA transforms preference-weighted objectives into constraints, adds existing safety constraints, and uses a linear approximation to update the policy without extra optimization variables.

    Source: §1 · p. 2

  • Results

    CoMOGA converges to a CP-optimal policy in tabular settings and achieves strong HV and SP performance with constraint satisfaction in experiments across tasks.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The experiments support avoiding gradient conflicts to prevent convergence to local optima, particularly in dynamically unstable locomotion tasks.

    Source: §1 · p. 2

    The convergence analysis assumes Slater’s condition and finite state and action spaces.

    Source: §4.4 · p. 5

Abstract

from arXiv · show

In many real-world applications, a reinforcement learning (RL) agent should consider multiple objectives and adhere to safety guidelines. To address these considerations, we propose a constrained multi-objective RL algorithm named Constrained Multi-Objective Gradient Aggregator (CoMOGA). In the field of multi-objective optimization, managing conflicts between the gradients of the multiple objectives is crucial to prevent policies from converging to local optima. It is also essential to efficiently handle safety constraints for stable training and constraint satisfaction. We address these challenges straightforwardly by treating the maximization of multiple objectives as a constrained optimization problem (COP), where the constraints are defined to improve the original objectives. Existing safety constraints are then integrated into the COP, and the policy is updated using a linear approximation, which ensures the avoidance of gradient conflicts. Despite its simplicity, CoMOGA guarantees optimal convergence in tabular settings. Through various experiments, we have confirmed that preventing gradient conflicts is critical, and the proposed method achieves constraint satisfaction across all tasks.

1 Introduction

Constrained multi-objective RL seeks feasible policies that are not dominated, allowing users to choose among trade-offs without additional training. CoMOGA addresses gradient conflicts and safety constraints, and experiments report stable constraint satisfaction across tasks.

1 Introduction

  • CMORL seeks a set of feasible, nondominated policies so users can select a preferred policy without additional training.Preferences encode the relative importance of objectives and can map to policies through a preference-conditioned policy.Source: §1 · p. 1
  • Linear scalarization is straightforward and converges to Pareto-optimal solutions in tabular settings, but with deep function approximators it tends toward local optima.The paper illustrates this issue by comparing linear scalarization and CoMOGA trajectories from different initial points.Source: §1 · p. 1, Figure 2 · p. 2
  • Treating constraints as objectives and adjusting their preferences aligns with the Lagrangian method, whose concurrent policy and multiplier updates can destabilize training.The introduction motivates handling constraints without additional optimization variables.Source: §1 · p. 2
  • CoMOGA treats multiple-objective maximization as a constrained optimization problem, converting objectives into preference-weighted constraints and incorporating existing safety constraints without extra optimization variables.A linear approximation computes the policy gradient while preventing the given objectives from decreasing; tabular convergence to a CP-optimal policy is also demonstrated.Source: §1 · p. 2
  • Experiments across diverse environments report constraint satisfaction on all tasks and support the effectiveness of avoiding gradient conflicts to prevent convergence to local optima.The authors note this is particularly relevant to unstable bipedal and humanoid locomotion, where suboptimal behaviors can include frequent falling.Source: §1 · p. 2

2 Related Work

Related MORL work includes linear and nonlinear reward scalarization, preference-based action distributions, and evolutionary strategies. Constrained RL methods differ in whether they solve a dual problem with multipliers or address constraints directly.

Multi-Objective RL

  • MORL methods use linear scalarization with Bellman operators or actor-critic approaches, while nonlinear Chebyshev scalarization can cover a non-convex policy set.Evolution strategies have also been applied to multi-objective optimization, including methods that update policy populations using linear scalarization.Source: §2 · p. 2
  • An alternative to reward-space scalarization applies preferences in the action-distribution space, and this approach was extended to CMORL.Source: §2 · p. 3

Constrained RL

  • Constrained RL includes primal-dual methods that update Lagrange multipliers and primal methods that address constraints directly without extra optimization variables.Primal approaches described here include trust-region, natural policy gradient, and multi-constraint methods; multiplier-based training can be unstable.Source: §2 · p. 3

3 Background

The background formalizes constrained multi-objective RL using reward objectives, cost constraints, and a set of constrained-Pareto-optimal policies. Preferences specify objective importance for a universal policy intended to cover the constrained-Pareto front.

Constrained Multi-Objective MDP

  • A constrained multi-objective MDP is defined by state and action spaces, transitions, vector rewards and costs, an initial-state distribution, and a discount factor.The reward and cost functions are bounded by Rmax.Source: §3 · p. 3
  • A constrained-Pareto-optimal policy is feasible and not dominated by another feasible policy; the set of such policies forms the constrained-Pareto front.Constrained dominance compares policies by whether one has no lower value on every reward objective.Source: §3 · p. 3
  • Gradient conflicts can cause convergence to local optima, motivating the search for the constrained-Pareto front while updating policies without such conflicts.Source: §3 · p. 3

Preference

  • A preference specifies the relative significance of objectives and conditions a universal policy mapping states and preferences to action distributions.The stated goal is to cover the constrained-Pareto front as extensively as possible.Source: §3 · p. 3

4 Proposed Method

CoMOGA transforms preference-weighted objectives and safety constraints into a constrained optimization problem, then aggregates gradients to update policies without objective conflicts. It also trains a universal policy across preferences, with CP-optimal convergence established under stated assumptions.

4.1 Transformation to Constrained Optimization

  • CoMOGA converts preference-weighted objective improvements and existing safety thresholds into constraints in a local optimization problem.The transformed problem minimizes the update’s H-norm while requiring objective improvements scaled by preference values.Source: §4.1 · p. 4

4.2 Gradient Aggregation

  • Linearizing the transformed problem yields a quadratic program whose aggregated gradient is clipped to keep the policy update within the local region.If the current policy violates a safety constraint, a recovery step solves a separate quadratic program using only safety constraints.Source: §4.2 · p. 4
  • The transformed constraints ensure the aggregated gradient does not conflict with any objective function.The process is illustrated in parameter space, where the objectives become constraints and the solution is scaled to remain local.Source: §4.2 · p. 4, Figure 3 · p. 5

4.3 Update Rule for Universal Policy

  • A replay-buffer loss combines policies for different preferences into one universal policy, which users can query with preferences after training without additional training.For each preference, the update targets an intermediate policy.Source: §4.3 · p. 5

4.4 Convergence Analysis

  • Under Slater’s condition and finite state and action spaces, the modified method converges to a constrained-Pareto optimal policy when H is the Fisher information matrix.The local-region size must satisfy the Robbins–Monro condition.Source: §4.4 · pp. 5–6

5 Practical Implementation

The practical implementation uses reward and cost critics to estimate objectives and constraints, then computes policy gradients through reparameterization.

5 Practical Implementation

  • Reward and cost critics estimate the objectives and constraint functions used for CoMOGA and the universal-policy update.The critics are trained by minimizing loss functions.Source: §5 · p. 6
  • The reward-critic target uses the immediate reward plus a discounted value estimate or Retrace(λ), and the same process applies to cost critics.The target value is denoted Y_Ri.Source: §5 · p. 6
  • Policy gradients are computed through the reparameterization trick, and the proposed method is summarized in Algorithm 1.Source: §5 · p. 6

6 Experiments

The experiments compare CoMOGA with existing constrained and adapted multi-objective methods across constrained and unconstrained tasks, using hypervolume and normalized sparsity to assess estimated constrained-Pareto fronts. Results cover legged locomotion, Safety Gymnasium, MO Gymnasium, and an ablation of the proposed approach.

6.1 Evaluation Metrics and Baselines

  • Evaluation measures estimated constrained-Pareto-front coverage with hypervolume and density with normalized sparsity, comparing CoMOGA against LP3 and Lagrangian-adapted PD-MORL and CAPQL.Normalized sparsity is intended to remove its correlation with hypervolume while preserving its ability to measure sparsity.Source: §6.1 · pp. 6–7

6.2 Legged Robot Locomotion

  • Across legged tasks, CoMOGA achieves the highest HV and lowest SP while satisfying all constraints, and surpasses baselines in bipedal locomotion.LP3 performs similarly in quadrupedal tasks but underperforms in bipedal tasks.Source: §6.2 · p. 7, Figure 4 · p. 8

6.3 Safety Gymnasium

  • In Safety Gymnasium, CoMOGA achieves the highest HV in every task, the lowest SP in two of four, and satisfies all constraints.LP3 and PD-MORL violate constraints in at least one task; CAPQL satisfies them but performs weakly on single-agent tasks.Source: Figure 5 · p. 8, §6.3 · p. 8

6.4 Multi-Objective Gymnasium

  • On MO Gymnasium, CoMOGA has the highest HV in three tasks and lowest SP in five of six, with outstanding performance on Hopper and Humanoid.All algorithms except LP3 show high HV on Walker2d, which the authors consider may reflect reaching the global optimum.Source: Figure 6 · p. 9, §6.4 · p. 9

6.5 Ablation Study

  • CAGrad combined with Lagrangian constraint handling violates the second quadrupedal constraint and has substantially lower HV and SP than CoMOGA on bipedal tasks.The comparison supports CoMOGA's ability to handle both constraints and gradient conflicts.Source: Figure 7 · p. 9, §6.5 · p. 9

7 Conclusions and Limitations

CoMOGA converts objectives into constraints to avoid gradient conflicts while handling safety constraints without additional optimization variables. It converges to a constrained-Pareto optimal policy in tabular settings, but trained policies may cover only a narrow portion of the constrained-Pareto front.

7 Conclusions and Limitations

  • CoMOGA converges to a constrained-Pareto optimal policy in tabular settings.Its experiments also report strong hypervolume and sparsity performance with constraint satisfaction.Source: §7 · p. 9
  • The objective-to-constraint transformation avoids gradient conflicts and handles safety constraints without additional optimization variables.Source: §7 · p. 9
  • Preventing objective decreases may leave trained policies covering only a narrow portion of the constrained-Pareto front.The authors suggest mitigating the gradient-conflict condition to better balance objectives and expand coverage.Source: §7 · p. 9

A Toy Example

The toy example illustrates how linear scalarization can be dominated by one objective, while the transformation proof establishes equivalence between two formulations under a linear assumption.

A Toy Example

  • The toy problem minimizes L1 and L2 subject to C(x1, x2) ≤ 0.The objective landscapes are shown in Figure 8.Source: A Toy Example · p. 13, Figure 8 · p. 13
  • For the toy example's initial point (−10, 7.5), linear scalarization can be dominated by L1 and converge to its local optimum.The authors state that avoiding gradient conflicts can help the policy converge to an optimal point.Source: A Toy Example · p. 13

B.1 Proof of Transformation Process

  • Under the linear assumption, the proof derives equivalent solutions for formulations (2) and (3).The resulting update is given as ϵH^-1gi/… in the supplied passage.Source: §B.1 · p. 14
  • The proof concludes that both formulations yield the same solution.Source: §B.1 · p. 14

B.2 Proof of Theorem 4.2

The proof establishes that the specified policy-update rule converges to a constrained-Pareto optimal policy. It treats feasible and constraint-violating updates separately and relates the limiting policy to linear-scalarization optimality.

B.2 Proof of Theorem 4.2

  • The convergence argument uses an update rule with gradients of objectives and constraints, a Robbins–Monro step size, and the Fisher information matrix pseudo-inverse.Source: §B.2 · p. 15
  • The proof handles both cases: all constraints are satisfied, or constraints are violated.It sums inequalities across time steps to establish the limiting-policy result.Source: §B.2 · pp. 15–17
  • The proof concludes that the policy converges to a constrained-Pareto optimal policy.The argument derives a sequence of feasible time steps whose limiting policy is linear-scalarization optimal for a preference, then applies Lemma B.1.Source: §B.2 · pp. 15, 17

B.3 Proof of Theorem 4.3

Theorem 4.3 establishes convergence under conditions on the update sequences. The appendices also specify the experimental tasks and metrics, and describe a Lagrangian baseline that parameterizes multipliers for continuous preferences.

B.3 Proof of Theorem 4.3

  • Theorem 4.3 states that the policy update converges to a constrained-Pareto optimal policy when ω is defined in (10).The proof constructs sequences satisfying Theorem 4.2's conditions; these include bounded multiplier terms and Robbins–Monro step sizes.Source: §B.3 · p. 18

C.1.1 Safety Gymnasium

  • Safety Gymnasium tasks cover single- and multi-agent navigation with point and car robots, objectives, and collision-avoidance constraints.Single-agent tasks add energy consumption as a second objective; multi-agent tasks use two goal objectives and two hazard constraints.Source: §C.1.1 · p. 19

C.1.2 Legged Robot Locomotion

  • Legged-robot tasks optimize command tracking and energy use while retaining three constraints from the original setup.The tasks include bipedal and quadrupedal robots; the bipedal energy objective penalizes action because valid torque information is difficult to obtain.Source: §C.1.2 · p. 20

C.1.3 Multi-Objective Gymnasium

  • The unconstrained Multi-Objective Gymnasium evaluation uses six continuous-action MuJoCo tasks with between two and three objectives each.Source: §C.1.3 · pp. 20–21

C.2 Metric Details

  • The estimated constrained-Pareto front is built from non-dominated objective values for preferences whose constraint values meet their thresholds.The method samples equally spaced preferences and rolls out the trained universal policy for each.Source: §C.2 · p. 21
  • Hypervolume measures front coverage relative to a reference point derived from the combined results of all algorithms.Normalized sparsity averages squared distances in normalized objective space to measure distribution while reducing its correlation with hypervolume.Source: §C.2 · p. 21, Figure 11 · p. 21

C.3 Implementation Details

  • The experiments report training time for the single-point goal task and use a PC with an Intel Xeon E5-2680 CPU and NVIDIA TITAN Xp GPU.Source: §C.3 · p. 22, Table 2 · p. 22

C.3.1 MORL to CMORL using the Lagrangian method

  • The Lagrangian baseline represents multipliers as a neural network of preference ω and updates policy parameters and multipliers concurrently.The multipliers are learned separately for each preference in principle, but preferences are continuous.Source: §C.3.1 · p. 22

C.3.2 Network Architecture

The methods use actor-critic networks, with distributional critics for Safety-Gymnasium and locomotion tasks and standard critics for unconstrained MO-Gymnasium tasks. Hyperparameter settings for constrained and unconstrained tasks are reported separately.

C.3.2 Network Architecture

  • The proposed method and baselines use fully connected policy and critic networks, while Lagrangian baselines also use multiplier networks.Their network structures are presented in Table 3.Source: §C.3.2 · p. 22, Table 3 · p. 23
  • Quantile distributional critics estimate objectives or constraints by averaging quantiles for Safety-Gymnasium and locomotion, while MO-Gymnasium uses scalar-value critics.TD(λ) trains distributional critics for the proposed method and LP3; PD-MORL and CAPQL minimize Wasserstein distance to truncated one-step TD targets.Source: §C.3.2 · pp. 22–23
  • Policy networks output the mean and diagonal variance of a normal distribution, but locomotion policies fix the variance and output only the mean.The variance is fixed to lower training difficulty.Source: §C.3.2 · p. 23

C.3.3 Hyperparameter Settings

  • Hyperparameter settings for Safety-Gymnasium and locomotion are reported in Table 4, and settings for MO-Gymnasium are reported in Table 5.These correspond to the CMORL and MORL tasks, respectively.Source: §C.3.3 · p. 23, Table 4 · p. 23, Table 5 · p. 24

D Additional Experimental Results

This section visualizes estimated CP fronts for two-objective tasks across legged locomotion, Safety Gymnasium, and MO Gymnasium. The fronts use 20 equally spaced preferences and are estimated from five random seeds.

Estimated CP Fronts

  • Figures 12–14 visualize estimated CP fronts for legged robot locomotion, Safety Gymnasium, and MO-Gymnasium tasks; in Figure 12, each task’s estimated front is plotted from five random seeds.The fronts are obtained by rolling out a trained universal policy with 20 equally spaced preferences; MO-Gymnasium hopper and ant tasks are excluded because they have three objectives.Source: D Additional Experimental Results · p. 24, Figure 12 · p. 24, Figure 13 · p. 25

Related papers

Loading 2403.00282v2…