Source-linked AI summary
Dynamic Weights in Multi-Objective Deep Reinforcement Learning
Axel Abels, Diederik M. Roijers, Tom Lenaerts, Ann Nowé, Denis Steckelmacher
TL;DR
The paper studies multi-objective RL when objective weights change over time, a setting where existing tabular approaches do not extend to high-dimensional inputs. It proposes a weight-conditioned Q-network with Diverse Experience Replay and reports dominance over adapted methods across weight-change scenarios and domains.
Problem
Dynamic objective weights require online multi-objective RL methods, while prior tabular approaches are infeasible for high-dimensional inputs.
Method
The paper proposes a Conditioned Network producing weight-dependent multi-objective Q-values and Diverse Experience Replay that preserves diverse trajectories across weight regions.
Results
CN dominates adapted algorithms across weight-change scenarios, and DER improves performance across tested algorithms.
Takeaways & Limitations
Weight-conditioned value learning can generalize across weight vectors, while diverse trajectory replay supports learning under changing weights.
Takeaways & Limitations
The multi-network approach requires long training times for each weight vector because inaccurate Q-values bias policy selection toward overestimated policies.
Abstract
from arXiv · showhide
Many real-world decision problems are characterized by multiple conflicting objectives which must be balanced based on their relative importance. In the dynamic weights setting the relative importance changes over time and specialized algorithms that deal with such change, such as a tabular Reinforcement Learning (RL) algorithm by Natarajan and Tadepalli (2005), are required. However, this earlier work is not feasible for RL settings that necessitate the use of function approximators. We generalize across weight changes and high-dimensional inputs by proposing a multi-objective Q-network whose outputs are conditioned on the relative importance of objectives and we introduce Diverse Experience Replay (DER) to counter the inherent non-stationarity of the Dynamic Weights setting. We perform an extensive experimental evaluation and compare our methods to adapted algorithms from Deep Multi-Task/Multi-Objective Reinforcement Learning and show that our proposed network in combination with DER dominates these adapted algorithms across weight change scenarios and problem domains.
1. Introduction
The paper addresses multi-objective RL when linearization weights change over time, especially in high-dimensional settings where tabular methods are infeasible. It proposes a weight-conditioned Q-network and Diverse Experience Replay, evaluating them across weight-change scenarios.
- Problem setting: Dynamic weights arise when linear scalarization remains applicable but objective weights change over time.For example, increasing fuel costs can make shorter travel time less worthwhile.
- Motivation: High-dimensional inputs such as camera images make tabular RL methods unsuitable for complex real-world problems.Deep RL supports image-based inputs, but most prior work focuses on single-objective problems.
- Contributions: The Conditioned Network outputs weight-dependent multi-objective Q-value vectors for high-dimensional dynamic-weight problems.Its training uses an update rule specific to the dynamic weights setting.
- Contributions: Diverse Experience Replay maintains diverse trajectories to improve sample efficiency and reduce replay-buffer bias.The buffer is designed to preserve experiences relevant across changing objective weights.
- Evaluation: Minecart is introduced as a high-dimensional benchmark requiring visual learning, adaptation to changing resource valuations, and fuel minimization.The task evaluates multi-objective behavior directly from raw visual input.
- Results: Across two weight-change scenarios, only CN both adapts quickly to sparse abrupt changes and converges to optimal policies under regular changes.DER further improves the performance of all tested algorithms by maintaining diverse trajectories.
2. Background
The background introduces MDPs, Q-learning, DQN, and multi-objective RL, then distinguishes offline methods from online dynamic-weight settings. It motivates online approaches because changing weights and cumulative regret make pre-training a full policy set inadequate.
- Markov Decision Processes: An MDP specifies states, actions, probabilistic transitions, and immediate rewards, with discounted cumulative reward as the optimization objective.The discount factor γ determines how future rewards contribute.
- Markov Decision Processes: A trajectory is a sequence of transitions connecting an initial state to a later state.Its transitions include states, actions, rewards, and successor states.
- Q-learning: Q-learning updates state-action values from observed transition tuples toward rewards plus discounted next-state estimates.Under stated assumptions, Q-learning converges.
- Deep Q-learning: DQN extends Q-learning to high-dimensional environments by approximating the Q-function with a neural network and training from replayed experiences.A fixed target network limits rapid propagation of approximation errors, while replay improves sample efficiency and reduces correlation.
- Multi-objective RL: Multi-objective MDPs use vector-valued rewards, and linear scalarization combines objective values using a weight vector.A convex coverage set contains an optimal policy for every linear scalarization.
- Offline and online MORL: Offline MORL can approximate a convex coverage set with multiple Q-networks trained sequentially on corner weights.This approach presumes sufficient pre-training time before deployment.
- Offline and online MORL: Online dynamic-weight MORL is challenging because weights vary over time and cumulative regret evaluates performance against the optimal policy.Pre-training an entire convex coverage set is described as inadequate in this setting.
- Related approaches: Universal Value Function Approximators generalize value functions across multiple goals by conditioning a single network on goal information.The paper adapts this idea for weight-conditioned multi-objective Q-values.
3. Contributions
The paper proposes CN and DER for dynamic-weight deep multi-objective RL, alongside Minecart as a high-dimensional benchmark for changing resource valuations.
- Conditioned Network (CN): CN conditions a multi-objective Q-network on weight vectors to generalize Q-value vectors across regions of weight space.The network processes raw visual features and feeds a weight input into an N-objective Dueling DQN head.
- Conditioned Network (CN): CN trains each experience using both the active weight vector and a randomly sampled previously encountered weight vector.This update rule promotes convergence on the current policy while preserving previously learned policies.
- Conditioned Network (CN): CN differs from UVFA through multi-objective outputs, end-to-end training, and an update rule adapted to dynamic weights.These design choices are presented as the method’s three major deviations from UVFA.
- Diverse Experience Replay: DER replaces recency-based replay with diversity-based memorization over trajectories, retaining experiences relevant to different weight-space regions.Trajectories are treated as atomic units and evaluated using return-vector signatures and crowding distance.
- Minecart Problem: Minecart provides a continuous-state, stochastic, delayed-reward visual benchmark in which agents mine resources, sell them, and minimize fuel under changing valuations.Its default weight space contains seven regions with different optimal policies.
- Minecart Problem: The benchmark evaluates adaptation to changing weights, while DER maintains trajectories with diverse future rewards for replay.The Minecart setting includes sparse resource rewards and delays between actions and resulting rewards.
4. Adapted Algorithms
The adapted algorithms include UVFA and a multi-network approach, but related methods are inadequate or suboptimal for dynamic weights because they require extensive pretraining or adaptation.
- Overview: Related deep multi-objective RL methods are suboptimal when adapted to dynamic weights.The paper presents these adaptations for completeness before introducing its own methods.
- UVFA: The UVFA adaptation concatenates state features with a weight vector and trains scalar Q-values end-to-end without expensive pre-training.Because distance-based goal metrics do not apply, the weight vector is directly combined with state features.
- Multi-Network (MN): The multi-network approach gradually builds a set of policies represented by MO Q-networks covering different regions of weight space.It stores only undominated networks, expected to keep the maintained set relatively small because the CCS is typically small.
- Multi-Network (MN): For each new weight vector, MN hot-starts from the saved policy with the highest scalarized value for that vector.The selected policy’s Q-network parameters are copied into the current policy’s Q-network.
- Multi-Network (MN): MN can require long training times because inaccurate predicted Q-values bias policy selection toward overestimated policies.The comparison between policies depends on accurate value estimates before adaptation can proceed reliably.
5. Experimental Evaluation
The evaluation tests dynamic-weight algorithms on Minecart and image-based DST under sparse and regular weight changes, using cumulative regret across repeated runs. CN generally performs best, while DER improves performance when replay diversity is needed.
- Experimental setup: Experiments cover Minecart and image-based DST under sparse changes every 50k or 5k steps and regular changes over 10 episodes.Results use 10 runs, smoothed over 200 steps, and evaluate cumulative regret, where lower regret is better.
- Evaluation metric: Cumulative regret measures the difference between the optimal value for the current weight vector and the agent’s actual return.This metric gives every weight vector the same optimal reference, so an optimal policy has zero regret.
- Sparse weight changes: DER maintains trajectories spread across return space, countering replay-buffer bias toward recent weights and supplying experiences for less recently executed policies.In Minecart, the diverse buffer’s blue trajectories span possible returns, unlike the recent-experience buffer’s orange concentration.
- Sparse weight changes: CN learns faster than MN while both ultimately learn strong policy sets, and CN with DER outperforms alternative conditioned-network training procedures.CN-UVFA trains uniformly over encountered weights and converges more slowly without DER; DER reduces the impact of missing relevant experiences.
- DST: CN+DER performs best on DST, although DER has no significant impact on MN and CN-ACTIVE and MN perform relatively worse than on Minecart.The authors hypothesize that smaller distances between DST’s optimal policies are harder to distinguish from approximation errors.
- Regular weight changes: Under regular changes, CN remains close to optimal across weight vectors, whereas MO converges to one policy, MN lacks time to converge, and CN-UVFA remains poor.CN continuously trains one network toward multiple policies, while emphasizing the current weight remains important in Minecart.
- Overall findings: CN dominates the other algorithms across the tested settings, while DER improves performance when trajectory diversity does not arise naturally.MN performs well only when it has enough training time to learn accurate Q-values.
6. Related Work
Prior work established dynamic weights for low-dimensional tabular RL, while related deep methods address different settings or require policy sets built in advance. This paper adapts these ideas to online dynamic weights with high-dimensional inputs.
- Tabular dynamic weights: Natarajan and Tadepalli solve dynamic weights for low-dimensional problems by training a set of policies with tabular RL.The cited tabular approach is not representative of the paper’s continuous, high-dimensional benchmark setting.
- Deep and offline MORL: MN extends similar policy-memory ideas to Deep RL, while DOL addresses offline MORL with unknown weights rather than online MORL with changing weights.DOL uses a convex coverage set whose policies are implemented by DQNs, but its problem has a small underlying state space.
7. Conclusion and Future Work
The paper concludes that CN handles high-dimensional dynamic-weight problems effectively, while DER addresses replay bias; future work targets broader transfer and replay methods.
- CN learns weight-dependent multi-objective Q-values for high-dimensional dynamic-weight problems.
- DER maintains diverse trajectories so policies can benefit from experiences beyond the recent-weight distribution.
- CN dominates adapted algorithms across different weight-change scenarios and approaches optimality on Minecart and DST.
- MN fails to converge when weights change regularly, whereas CN remains close to optimality on Minecart and DST.
- Future Work: Future work will add transfer-learning techniques and explore variants of DER.
1. Algorithms
The algorithms address dynamic weights through either a single weight-conditioned network or multiple reusable networks, with DER improving replay diversity.
- Conditioned Network: CN conditions one Q-network on the current weight vector to output weight-dependent multi-objective Q-values.
- Conditioned Network: CN trains on both current and randomly sampled historical weight vectors to generalize across the relevant weight simplex.
- Conditioned Network: CN priorities use the average TD-error from the two weight vectors on which each experience was trained.
- Multi-Network: MN gradually builds an approximate partial convex coverage set by training a network for each encountered weight vector and reusing policies.
- Diverse Experience Replay: DER combines a FIFO replay buffer with a secondary diverse buffer that considers evicted trajectories for memorization.
2. Implementation details
The implementation uses a multi-objective dueling architecture that extracts visual features and produces action-specific objective values, with CN additionally receiving weights.
- Network Architecture: The network uses additional dense layers to represent multi-objective Q-values and support precise or weight-conditioned policy learning.
- Hyperparameters: Table 2 provides the optimization hyperparameters used in the experiments.
- Network Architecture: The advantage stream outputs |A|×N units, the value stream outputs N units, and their combination produces |A|×N Q-values.
- Network Architecture: Convolutional and fully connected layers extract features before feeding a Multi-Objective Dueling DQN head.
- Conditioned Network: For CN, the weight vector is added as an input parameter to the network functions and enters the Q-value head.
3. Test Problems
The evaluation uses Minecart and modified DST to test dynamic-weight learning in visual environments with multiple objectives, stochasticity, delayed rewards, and policy regions.
- Minecart Problem: Minecart is a continuous, stochastic environment with delayed rewards for resource collection and fuel minimization.
- Minecart Problem: Minecart agents navigate visual maps, collect ore, return to base, and sell resources using movement and mining actions.
- Minecart Problem: Minecart reward vectors contain resource amounts and consumed fuel, with sparse resource rewards and delayed consequences.
- Deep Sea Treasure: The modified DST map gives each treasure an optimal policy in the convex coverage set at γ = 0.95, with approximately 10% of weight vectors favoring each policy.
4. Additional Results
Additional experiments compare replay strategies and a naive scalarization baseline across dynamic-weight scenarios. DER generally outperforms selective replay, exploration-based selection harms performance, and the naive algorithm fails to learn trade-offs.
- Weight-dependent exploration metrics are unreliable because an action can be exploratory for one weight vector but optimal for another.
- Trajectory-level replay preserves the action sequences leading to rewarding experiences, unlike transition-based selection that may retain only the final interesting transition.
- Table 5 reports average episodic regret and improvement over the MO with Std. ER baseline for DST, separately for overall performance and the last 25k steps.
- Selective experience replay performs worse than DER in the sparse weight-change scenario, although it generally improves over standard experience replay.
- Exploration-based selection consistently damages performance, including in both sparse and regular weight-change scenarios.
- The naive algorithm performs poorly across experiments because it fails to learn trade-offs between objectives, despite performing well for edge weight vectors.