Source-linked AI summary
Multi-Objective Deep Reinforcement Learning
Hossam Mossalam, Yannis M. Assael, Diederik M. Roijers, Shimon Whiteson
TL;DR
Multi-objective reinforcement learning must identify policies across unknown objective trade-offs, a setting that deep MDP methods had not addressed. The paper combines deep Q-learning with optimistic linear support to learn approximate coverage sets, finding that partial network reuse performs best in its experiments and that DOL can learn accurate CCSs for large-input problems. The approach remains limited because reinforcement-learning approximation quality lacks the error bounds used by OLS in planning.
Problem
Deep learning methods for MDPs had not been extended to MOMDPs, where unknown preferences require a coverage set rather than a single optimal policy.
Method
DOL combines an OLS outer loop with scalarised deep Q-learning that outputs vector-valued Q-values, with DOL-FR and DOL-PR adding full or partial network reuse.
Results
DOL-PR outperforms DOL and DOL-FR on large-input problems, while the algorithms learn CCSs with high accuracy; DOL-PR achieves the least error in Multi-Objective Mountain Car.
Takeaways & Limitations
Partial reuse is the preferred algorithm because it preserves learned state-space features while allowing new Q-values to be trained from scratch.
Takeaways & Limitations
Unlike planning-based OLS, reinforcement learning provides no upper bounds on deep Q-learning approximation quality because the true CCS is unknown.
Abstract
from arXiv · showhide
We propose Deep Optimistic Linear Support Learning (DOL) to solve high-dimensional multi-objective decision problems where the relative importances of the objectives are not known a priori. Using features from the high-dimensional inputs, DOL computes the convex coverage set containing all potential optimal solutions of the convex combinations of the objectives. To our knowledge, this is the first time that deep reinforcement learning has succeeded in learning multi-objective policies. In addition, we provide a testbed with two experiments to be used as a benchmark for deep multi-objective reinforcement learning.
1 Introduction
Deep reinforcement learning had advanced mainly in single-objective settings, whereas multi-objective problems require policies covering unknown trade-offs. The paper introduces DOL and reuse variants to learn approximate coverage sets for high-dimensional MOMDPs.
- Motivation: Multi-objective problems involve potentially conflicting objectives whose trade-offs are often unknown a priori, so no single policy is optimal for every utility function.A coverage set should contain at least one optimal policy and value vector for each possible utility function.
- Research gap: Deep learning methods for MDPs had not previously been extended to MOMDPs because neural networks must handle unknown preferences and sets of value vectors.The paper addresses this by evaluating a sequence of scalarised single-objective problems within an outer-loop framework.
- Approach: DOL combines an OLS outer loop with an OLS-compliant neural-network solver to learn an approximate coverage set of policies.OLS repeatedly calls a single-objective solver and produces an approximate coverage set after finitely many calls.
- Contributions: DOL-PR reuses part of the previous network and empirically outperforms full reuse, drastically improving performance over DOL without reuse.The reuse variants exploit increasing similarity between successive scalarised problems and their optimal value vectors.
2 Background
MOMDPs extend reinforcement learning to vector-valued rewards and require coverage sets across user preferences. OLS constructs such sets by iteratively solving scalarised MDPs, but reinforcement-learning error bounds are unavailable and must be replaced by a heuristic.
- Deep Q-Networks: Standard deep Q-learning represents action values with neural networks, uses frozen target networks and experience replay, and typically selects actions with an ϵ-greedy policy.The target network remains frozen while the online network is updated by gradient descent.
- Multi-Objective MDPs: In an MOMDP, each transition produces a vector of rewards, and the solution is a coverage set containing an optimal policy for each possible preference.The paper focuses on linear scalarisation functions that map policy value vectors to scalar values.
- Optimistic Linear Support: OLS incrementally constructs an approximate coverage set by solving scalarised single-objective MDPs for different linear weight vectors.Each iteration adds a policy value vector to the intermediate set and prioritises unexplored corner weights by estimated improvement.
- Optimistic Linear Support: The scalarised value function over a set of value vectors is piecewise linear and convex, with corner weights marking its convex upper surface.OLS selects corner weights using an optimistic upper-bound criterion and solves the corresponding scalarised MDP.
- Reinforcement-learning limitation: In reinforcement learning, the true CCS is unknown and deep Q-learning provides no upper bounds on approximation quality, so OLS uses V∗S(w) as a heuristic priority criterion.The CCS definition assumes that values found for previous weights were optimal for those weights.
3 Methodology
The methodology adapts DQN to OLS by learning vector-valued action values under linear scalarisation, then iteratively expanding a partial coverage set. DOL-FR and DOL-PR reuse previous networks to accelerate successive scalarised problems.
- Deep OLS Learning (DOL): DOL requires an OLS-compliant learner that outputs vector-valued Q-values accurate enough to select the next corner weight.Its network outputs a |A| × n matrix rather than one value per action and is trained for more episodes.
- Deep OLS Learning (DOL): Scalarised deep Q-learning maximises the inner product of the weight vector and Q-values, and using it inside OLS yields DOL.This preserves the single-objective DQN solver while making its outputs suitable for multi-objective OLS.
- Reuse variants: The reuse variants exploit similarity between nearby scalarisation weights because their optimal value vectors and policies are typically similar.The nearest previously learned network is selected for reuse at the current corner weight.
- Algorithm design: Algorithm 1 implements DOL, DOL-FR, and DOL-PR by setting the reuse parameter to none, full, or partial.The algorithm also manages explored weights, stored DQNs, improvement thresholds, and the partial coverage set.
- Reuse variants: DOL-FR copies all parameters from the nearest network, whereas DOL-PR copies the network but randomly reinitialises its last layer to escape local optima.DOL without reuse reinitialises all network parameters.
- Deep OLS Learning (DOL): DOL maintains a partial CCS, explores corner weights in priority order, and adds a value vector only when it improves the current set.Obsolete corner weights and vectors that are no longer optimal for any weight are removed.
4 Experimental Evaluation
The evaluation tests DOL and its reuse variants on multi-objective Mountain Car and Deep Sea Treasure, using raw states and image inputs. The results show rapid CCS approximation on Mountain Car, strong performance from partial reuse on Deep Sea Treasure, and sensitivity to training duration.
- Experimental setup: DOL, DOL-FR, and DOL-PR are evaluated on Mountain Car and Deep Sea Treasure with direct state inputs, then tested for scalability using an image-based DST variant.The experiments use DQN-style experience replay, target networks, epsilon-greedy exploration, and neural architectures matched to raw or image inputs.
- Multi-Objective Mountain Car: Mountain Car adds fuel consumption as a second objective, producing a small CCS with two value vectors.The agent must balance reaching the goal against fuel use, which is proportional to exerted force.
- Multi-Objective Mountain Car: DOL-PR achieves the least MC CSS error, while all three algorithms learn a good CCS approximation within 2 iterations.Further improvements after two iterations are tiny, and reuse offers little benefit because the first extrema policies are very different.
- Deep Sea Treasure: Deep Sea Treasure uses a 10 × 11 grid, ten treasures, and two objectives: treasure value and a −1 time penalty per step.Treasure values are adapted so the most efficient policy for reaching each treasure lies in the CCS, and both rewards are normalised between [0, 1].
- Deep Sea Treasure: DOL-PR exhibits the highest DST performance because resetting the last layer preserves the encoded state space while training new Q-values from scratch.The evaluation concludes that DOL-PR is the preferred algorithm for this setting.
- Accuracy vs Episodes: DOL-PR lacks sufficient accuracy with few episodes, improves through 4000 episodes, and overfits at 10000 episodes, reducing performance.The episode-count study measures maximum CCS error.
5 Related Work
Related MORL work primarily uses inner-loop value-vector methods or heuristic policy search, while outer-loop approaches generally lack deep reinforcement learning. The paper positions deep RL as important for scaling multi-objective methods to larger problems.
- Multi-objective reinforcement learning: Most MORL algorithms use an inner-loop approach that replaces single-objective solvers’ value calculations with sets of value vectors.The paper argues this approach is fundamentally different from its outer-loop design and is not clearly applicable to DQN back-propagation.
- Heuristic policy search: Heuristic policy-search methods use multi-objective evolutionary algorithms or Pareto local search to find alternative policies.Although evolutionary methods can work with neural networks, neural-network evolution is typically slower than back-propagation.
- Outer-loop methods: Other outer-loop MORL methods do not employ deep reinforcement learning, whereas OLSAR applies reuse to alpha matrices in multi-objective POMDP planning.OLSAR’s alpha matrices provide a guaranteed lower bound and can be fully reused without affecting necessary exploration, unlike the setting considered here.
6 Discussion
The paper’s three algorithms extend deep Q-learning to multiobjective reinforcement learning by learning policies and value vectors iteratively. Empirically, they learn high-accuracy convex coverage sets on large-input problems, with partial parameter reuse outperforming the alternatives.
- The three algorithms enable deep Q-learning for multiobjective reinforcement learning by learning one policy and corresponding value vector per iteration.
- DOL builds on the optimistic linear support framework, while DOL-FR and DOL-PR reuse neural-network parameters between iterations to accelerate learning.
- DOL-PR outperforms DOL and DOL-FR on large-input problems, indicating that parameter reuse helps and partial reuse avoids remaining stuck in a policy optimal for a previous w.
- The algorithms learn convex coverage sets with high accuracy on problems with large inputs.