Source-linked AI summary
A Multi-Objective Deep Reinforcement Learning Framework
Thanh Thi Nguyen, Ngoc Duy Nguyen, Peter Vamplew, Saeid Nahavandi, Richard Dazeley, Chee Peng Lim
TL;DR
Existing MORL methods face difficulty with high-dimensional state spaces and limited standardized support for deep methods. This paper introduces a modular, scalable DQN-based MODRL framework supporting single- and multi-policy strategies with linear and non-linear action selection. Experiments on Deep Sea Treasure and MO-Mountain Car indicate effective discovery of Pareto-optimal solutions, while multithreading reduces multi-policy training requirements.
Problem
Existing MORL methods are difficult to extend to complex or high-dimensional environments, and no standard benchmark framework supports deep-learning algorithms such as DQN.
Method
The paper develops a modular DQN-based MODRL framework supporting single- and multi-policy strategies, linear and non-linear action selection, and scalar, vector, or graphical state representations.
Results
The framework effectively finds Pareto-optimal solutions on Deep Sea Treasure and three-objective Mountain Car, while multi-policy training requires 200,000 rather than 1,200,000 steps.
Takeaways & Limitations
The framework provides a testbed for comparing MODRL methods and accommodates different deep reinforcement learning algorithms for increasingly complex problems.
Takeaways & Limitations
The existing MORL_Glue benchmark supports vector state information but not images or deep-learning algorithms such as DQN.
Abstract
from arXiv · showhide
This paper introduces a new scalable multi-objective deep reinforcement learning (MODRL) framework based on deep Q-networks. We develop a high-performance MODRL framework that supports both single-policy and multi-policy strategies, as well as both linear and non-linear approaches to action selection. The experimental results on two benchmark problems (two-objective deep sea treasure environment and three-objective Mountain Car problem) indicate that the proposed framework is able to find the Pareto-optimal solutions effectively. The proposed framework is generic and highly modularized, which allows the integration of different deep reinforcement learning algorithms in different complex problem domains. This therefore overcomes many disadvantages involved with standard multi-objective reinforcement learning methods in the current literature. The proposed framework acts as a testbed platform that accelerates the development of MODRL for solving increasingly complicated multi-objective problems.
1. Introduction
The paper addresses the limited scalability and deep-learning support of existing MORL methods by proposing a modular MODRL framework for more complex problem domains. The framework supports varied state representations, policy strategies, action-selection approaches, and benchmark evaluation.
- Existing MORL research has focused largely on simple gridworld tasks, while tabular Q-learning becomes impractical as state spaces grow.
- MORL_Glue provides benchmark environments and tabular or tile-coding algorithms but does not support image-based states or deep-learning algorithms such as DQN.
- The proposed Python framework supports single-policy and multi-policy MODRL strategies and linear and non-linear action selection.
- A network-configuration design accepts scalar, vector, and graphical state representations while enabling plug-and-play modification of DRL algorithms.
- The framework uses hypervolume evaluation and graphical environments as a testbed for comparing MODRL methods and supporting future implementations and applications.
2. MORL Methods and Deep Learning Extensions
MORL methods represent rewards as vectors and address conflicting objectives through trade-offs, while MODRL extends these methods with deep networks and flexible policy and action-selection strategies. The proposed framework separates neural networks, DRL algorithms, and environments to support modular implementation of linear and nonlinear approaches.
- MORL foundations: MORL represents rewards as vectors, and conflicting objectives require trade-offs evaluated through Pareto fronts.A Pareto front captures compromise solutions among objectives.
- Policy strategies: Single-policy methods find one solution with lower computational expense but require prior objective preferences, whereas multi-policy methods approximate multiple Pareto-optimal solutions at higher cost.Multi-policy methods expose trade-off information so users can select among varied preferences.
- Action selection: MORL action selection can use linear or nonlinear scalarisation of vector-valued rewards.The framework demonstrates linear weighted sums and nonlinear thresholded lexicographic ordering (TLO).
- Deep-learning extensions: Prior DQN extensions addressed multi-policy linear MORL sequentially or single-policy nonlinear MORL through a specific subsumption architecture.These approaches did not jointly provide the framework's broader combination of policy strategies and action-selection methods.
- Framework architecture: The proposed framework modularizes neural networks, DRL algorithms, and environments so different DRL methods can be applied across problem domains.Its architecture uses configurable networks and a policy-network module before the DRL agent interacts with the environment.
- Single-policy DQN: The linear DQN approach receives a vector reward and fixed weight vector, with the weights omitted from the DQN inputs because scalarisation is fixed.The network uses multiple output groups, one per objective, with each group containing nodes for possible actions.
3. Experiment Settings and Evaluations
The framework is evaluated on Deep Sea Treasure and MO Mountain Car, selected for their different objective counts. Hypervolume provides a common performance measure during and after learning without requiring the true optimal front.
- Benchmark environments: Deep Sea Treasure and MO Mountain Car are used because they have different numbers of objectives.The study evaluates a two-objective Deep Sea Treasure setting and a three-objective Mountain Car setting.
- DQN settings: DQN parameters are selected by grid search so that convergence occurs in the minimum number of steps.The settings are reported in Table 1.
- Evaluation metric: Hypervolume is used because it gives a single comparison value and does not require the true optimal front or its approximation.Larger hypervolume indicates better performance, and improvements in accuracy, extent, or diversity increase the value.
- Evaluation procedure: The study measures hypervolume both after convergence and during learning to compare algorithm performance over time.Online hypervolume is obtained by temporarily disabling exploration for one completed episode before learning resumes.
4. The Deep Sea Treasure (DST) Problem
The DST experiments evaluate the MODRL framework on parameterized two-objective environments, showing how linear and nonlinear DQN methods reach different Pareto solutions. The results also compare single-policy and multi-policy learning through convergence and hypervolume behavior.
- Environment: DST varies state-space size and treasure rewards while optimizing treasure value and search-time penalty.The agent receives a two-element reward vector and can use scalar or graphical state representations.
- Single-policy linear DQN: 100,000 steps yield solution (1, -3) with linear scalarisation weights [0.01, 0.99].The convergence plot tracks treasure and time-penalty objective rewards during training.
- Pareto fronts: The 3-column DST Pareto solutions are (1, -3), (26.25, -5), and (100, -7), while Fig. 5 depicts fronts for 3-column and 5-column environments.The experiments focus on the 3-column environment, with 5-column results reported in Appendix A.
- Single-policy nonlinear DQN: Linear scalarisation cannot find (26.25, -5) with positive weights, whereas nonlinear TLO converges to this and (100, -7).TLO uses thresholds of 13.63 and 63.13 for the first objective in the two demonstrations.
- Single-policy versus multi-policy: Multiple agents can train in parallel, with each agent learning an individual policy for selecting among alternatives when goals change.The framework compares linear and nonlinear TLO agents using both single-policy and multi-policy methods.
- Single-policy versus multi-policy: The nonlinear method finds all solutions on the concave Pareto front, while the linear method does not converge to (26.25, -5).The reported result indicates that the deep-learning extension preserves properties observed for traditional MORL algorithms.
5. The MO Mountain Car Problem
The three-objective Mountain Car environment evaluates time, backward-acceleration, and forward-acceleration penalties using image-based DQN inputs. Linear weights and nonlinear thresholds produce different objective trade-offs, while multi-policy training finds six solutions with fewer training steps than single-policy training.
- Problem setup: The Mountain Car task minimizes episode length, backward acceleration, and forward acceleration, with each objective receiving a penalty of -1 per corresponding action or time step.Episodes terminate after 100 steps or when the car reaches the goal.
- Problem setup: The DQN receives the entire environment image as its state representation rather than the accrued objective values.The implementation integrates deep convolutional layers into the traditional Q-learning approach.
- Single-policy linear DQN: Six linear weight sets prioritize different objectives, including time-only, acceleration-only, and balanced backward-forward acceleration cases.The evaluated sets are (1,0,0), (0.5,0.5,0), (0.5,0,0.5), (0,1,0), (0,0.5,0.5), and (0,0,1).
- Single-policy linear DQN: Low time weight can produce the null-action policy, while the (0,0.5,0.5) setting effectively optimizes both acceleration objectives.With no time penalty, the agent may avoid penalties for backward and forward acceleration by always choosing the null action.
- Single-policy nonlinear DQN: Different weights or thresholds optimize different objective subsets, with objective 1 optimally reaching -20 and objectives 2 and 3 optimally reaching 0.Table 2 summarizes the corresponding single-policy linear and nonlinear DQN rewards.
- Single-policy versus Multi-policy: Multi-policy linear and nonlinear methods find six solutions in 200,000 training steps, versus 1,200,000 for single-policy methods, while linear methods dominate nonlinear TLO methods in hypervolume.The nonlinear TLO experiments require selecting thresholds after examining Q-value ranges and testing alternatives.
6. Conclusions and Further Work
The paper presents a scalable, high-performance MODRL framework that combines deep reinforcement learning with single- and multi-policy strategies. Its modular design supports multiple DRL algorithms and problem domains, while multithreading reduces training time for multi-policy tasks.
- Conclusions: The proposed Python framework integrates DRL algorithms into MORL to address high-dimensional environments that tabular Q-learning cannot handle.The framework is presented as scalable and high-performance.
- Conclusions: Single-policy and multi-policy strategies are supported, with multithreading reducing training time for multi-policy tasks.The framework is designed to solve MORL problems efficiently.
- Further work: The modular framework can accommodate algorithms including DQN, Dueling DQN, A3C, Double DQN, and UNREAL across environments from gridworlds and Mountain Car to Atari and MuJoCo.These extensions are identified as future directions for expanding the framework.
- Applications: MODRL is positioned for real-world problems with conflicting objectives, including autonomous-vehicle performance versus energy consumption.The paper also identifies chemical-engineering optimization as an application setting.
Appendix A
On the five-column Deep Sea Treasure problem, the framework finds all five policies on the actual Pareto front. Linear scalarisation finds the first solution, while nonlinear TLO finds the remaining four.
- Deep Sea Treasure results: The five-column Deep Sea Treasure problem has five target Pareto-front solutions: (1, -3), (5, -5), (17, -7), (49, -10), and (100, -13).Figures A1–A4 illustrate convergence to these solutions.
- Method comparison: Linear scalarisation finds (1, -3), whereas nonlinear TLO finds the remaining four Pareto-front solutions.The two methods therefore contribute different solutions in this benchmark.
- Convergence: The learning process converges to (1, -3), (5, -5), (17, -7), (49, -10), and (100, -13).The convergence trajectories are shown in Figures A1, A2, and A3.
- Evaluation: Online hypervolume values are reported during learning for single-policy and multi-policy methods using both linear and nonlinear TLO approaches.Figure A4 presents the corresponding hypervolume histories.