Source-linked AI summary
A Practical Guide to Multi-Objective Reinforcement Learning and Planning
Conor F. Hayes, Roxana Rădulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Mathieu Reymond, Timothy Verstraeten, Luisa M. Zintgraf, Richard Dazeley, Fredrik Heintz, Enda Howley, Athirai A. Irissappane, Patrick Mannion, Ann Nowé, Gabriel Ramos, Marcello Restelli, Peter Vamplew, Diederik M. Roijers
TL;DR
Real-world decision problems involve multiple conflicting objectives, while much reinforcement learning and planning still relies on single objectives or simple scalarisation. This article provides a practical guide to modelling and solving multi-objective problems, illustrating how problem factors influence system design. In a wet-clutch application, policies from some utility functions were superior to those reported by the original scalar-reward study.
Problem
Most real-world decisions are multi-objective, but many agents optimise a single objective or combine objectives into scalar additive rewards.
Method
The article guides researchers and practitioners through modelling, solving, and designing multi-objective decision-making systems while keeping user utility central.
Results
Some policies produced by examining multiple utility functions in wet-clutch engagement were superior to those reported in the original scalar-reward study.
Takeaways & Limitations
Multi-objective planning and learning can support decision systems whose policies reflect user preferences across complex problems.
Takeaways & Limitations
Multi-objective value-vector estimates can have high variance or systematic bias, especially with neural-network function approximators.
Abstract
from arXiv · showhide
Real-world decision-making tasks are generally complex, requiring trade-offs between multiple, often conflicting, objectives. Despite this, the majority of research in reinforcement learning and decision-theoretic planning either assumes only a single objective, or that multiple objectives can be adequately handled via a simple linear combination. Such approaches may oversimplify the underlying problem and hence produce suboptimal results. This paper serves as a guide to the application of multi-objective methods to difficult problems, and is aimed at researchers who are already familiar with single-objective reinforcement learning and planning methods who wish to adopt a multi-objective perspective on their research, as well as practitioners who encounter multi-objective decision problems in practice. It identifies the factors that may influence the nature of the desired solution, and illustrates by example how these influence the design of multi-objective decision-making systems for complex problems.
1 Introduction
Real-world decisions typically involve multiple conflicting objectives, but many planning and reinforcement-learning systems reduce them to a single scalar reward. The article introduces a practical multi-objective perspective for modelling and solving these problems while accounting for preferences, explainability, and changing trade-offs.
- Real-world decisions commonly require simultaneous trade-offs among objectives such as energy production, irrigation deficits, flooding risk, treatment effectiveness, and side effects.
- Scalar reward engineering explores only selected scalarisations, so an acceptable policy may be arbitrarily far from optimal utility.
- Scalar value functions provide limited a posteriori explanations because they report aggregate value changes rather than objective-specific effects.
- Linear additive rewards cannot represent some non-linear human preferences without breaking reward additivity, potentially losing optimality.
- Explicitly multi-objective systems can accommodate changing preferences without retraining or updating whenever those preferences change.
- The article guides practitioners and researchers through modelling, solving, evaluating, and deploying multi-objective decision systems for practical problems.
2 Motivating examples of modelling complex problems with multi-objective approaches
The article motivates multi-objective modelling through travel, water management, military procurement, wind-energy, and other complex decision problems. These examples show how competing objectives, uncertainty, interacting agents, long horizons, and changing conditions shape the appropriate solution.
- Travel choices trade off time, cost, comfort, and reliability, while congestion and accidents introduce uncertainty from interactions with other users.
- Journey planning may require balancing objectives across repeated trips or within each trip, leading to different policy requirements.
- Water-reservoir operations must balance agricultural water supply against competing socio-economic objectives through dam-gate and lake-level control.
- Military procurement requires long-term planning because equipment production times, manufacturing pipelines, and future operating conditions vary substantially.
- Military plans must be revised as governments, priorities, international dynamics, technology, and expected operations change over time.
- Wind-turbine control balances power production against fatigue loads and accumulated damage, but the relationship between actions and maintenance costs remains complex.
- Multi-objective learning and planning have also been applied to domains including distributed computing, molecular design, cybersecurity, simulation, and scheduling.
3 Problem setting
A MOMDP represents decisions with vector-valued rewards, so policies are only partially ordered unless an objective-prioritizing utility function is supplied. The resulting solution framework therefore reasons over sets such as the undominated set, Pareto Front, coverage sets, and convex coverage sets.
- MOMDP formulation: A MOMDP extends an MDP with a vector-valued reward R that provides immediate feedback for each of d ≥2 objectives.The reward-vector length equals the number of objectives.
- Policies and value functions: Policies in a MOMDP map states to action-selection distributions, and their value functions are vectors rather than scalars.The policy value Vπ belongs to R^d.
- Policies and value functions: Because multi-objective value functions generally induce only a partial ordering, selecting an optimal policy requires additional information about objective priorities.A utility function can scalarise values and restore a total ordering, but this is not always possible, feasible, or desirable.
- Solution sets: Without user-utility information, the problem is described through sets of possibly optimal policies and value vectors rather than one uniquely optimal value vector.The undominated set contains policies optimal for at least one possible utility function.
- Solution sets: The Pareto Front contains non-dominated policies, while a coverage set retains policies that achieve maximal scalarised value for every utility function.A Pareto Coverage Set may retain only one policy for each repeated value vector.
- Solution sets: For positively weighted linear utilities, the convex hull contains policies maximizing a weighted sum for some weight vector, whereas a CCS can be sufficient for broader monotonic utilities when stochastic policies are allowed.With discrete deterministic policies, a finite CCS can serve as a basis for constructing every policy in a PCS, while the CH is typically infinite.
4 The utility-based approach
The utility-based approach derives the desired solution set from available information about user utility and permitted policy types. It then guides solution-concept selection, algorithm choice, and policy selection through a staged pipeline.
- User utility properties can alter the desired solution, available methods, and whether stable solutions exist.
- The approach collects information about utility and policy constraints before deriving a solution concept suited to the user’s requirements.
- The workflow selects or designs an algorithm for the solution concept, distinguishing between single-policy and multi-policy algorithms.
- Utility application depends on whether utility is derived from individual outcomes or average returns, yielding ESR or SER respectively.ESR applies utility before taking expectations; SER takes expectations before applying utility.
- Permitted stochastic, deterministic, stationary, or non-stationary policies must be chosen because policy types affect the appropriate solution.Stochastic policies can be strictly better than deterministic ones, although domain requirements may rule them out.
- The complete pipeline helps users select a policy from the algorithm’s solution set, with more elaborate selection methods needed for large or continuous sets.Gaussian processes and preference information are examples of methods for supporting selection from large solution sets.
5 Factors influencing the design of multi-objective systems
Multi-objective system design depends on problem factors such as utility knowledge, preference dynamics, policy permissions, and the required number of policies. These factors determine the solution concept and motivate scenario-specific alternatives to a priori scalarisation.
- Multi-objective domains introduce factors absent from single-objective problems, with important implications for agent design.
- 5.1 Scenarios requiring a multi-objective approach: Unknown or difficult-to-specify preferences make a priori scalarisation infeasible or undesirable, favoring coverage sets and user selection.Decision support keeps the utility function implicit, while unknown-utility planning can reveal it later.
- 5.1 Scenarios requiring a multi-objective approach: Known preferences make scalarisation possible and feasible, but it may remain undesirable when a priori scalarisation creates an intractable problem.
- 5.1 Scenarios requiring a multi-objective approach: Interactive decision support elicits user preferences during learning while the agent also learns about the environment.Users can rank potential solutions to improve the representation of their preferences.
- 5.1 Scenarios requiring a multi-objective approach: Dynamic preferences favor learning a finite set of policies that can cover changing utility functions without waiting for policy adaptation.Adapting a single policy after preferences change can incur a period of sub-optimal behavior.
- 5.2.1 Single versus multiple policies: The selected solution concept determines whether the system learns one policy or multiple policies, linking scenario factors to solution-set design.Unknown utility and decision support require multiple policies, whereas known utility can support a single policy.
6 The relationship with other problems
The paper relates multi-objective decision making to neighboring areas while warning against treating objectives as independent agents. It also connects explicit objectives to fairness, safety, explainability, and human-aligned AI.
- Connections to related problems: POMDPs are a superclass of MOMDPs under linear utility, allowing MOMDPs to inherit theoretical properties from POMDP research.The correspondence interprets linear utility weights as beliefs over a hypothetical true objective.
- Multi-objective versus multi-agent problems: Objectives should not be modelled as agents because voting rules and Nash equilibria lack guarantees about the user’s utility.Such mechanisms target trade-offs among agents, not potentially nonlinear trade-offs among objectives of one agent.
- Multi-objective versus multi-agent problems: Altruistic agents can treat other agents’ utilities as objectives, enabling fairness constraints and Lorenz-optimal policy sets.A Lorenz-optimal set can support later negotiation over which policy to execute.
- Connections to related problems: Auxiliary tasks, UVFA, and successor features can be viewed as restricted multi-objective problems, especially when goals or objectives correspond to states or state features.Successor features specifically use linear task weights over state features.
- Human-aligned agents: Multi-objective methods provide a mechanism for addressing safety, ethics, legal restrictions, explainability, and fairness in human-aligned AI.These concerns extend beyond maximizing a conventional reward and include multiple stakeholder interests.
7 Survey of multi-objective reinforcement learning and planning algorithms
This section reviews multi-objective planning and reinforcement-learning algorithms in relation to application design factors. Its purpose is to help identify algorithms suited to particular application properties.
- Survey scope: The survey organizes existing planning and reinforcement-learning algorithms around design factors identified earlier in the paper.The organization is intended to support algorithm selection for particular applications.
- Survey scope: The review focuses on how algorithm properties relate to the requirements of multi-objective applications.It is framed as guidance for matching extant methods to application characteristics.
- Survey scope: The section aims to aid researchers and practitioners in identifying which extant algorithms may be best suited to a particular application.Suitability is assessed based on the properties of that application.
7.1 Multi-objective planning algorithms
Multi-objective planning spans early Pareto-set dynamic programming, methods for stochastic MOMDPs, convex-hull value iteration under linear utility, and specialized approaches for nonlinear utility.
- Foundations: Planning research on MOMDPs dates to the early 1980s and includes dynamic-programming methods for finding Pareto-set policies.Early approaches addressed infinite-horizon discounted MOMDPs but raised computational-feasibility concerns.
- Stochastic planning: Under stochastic transitions and SER, local state information is insufficient because optimal actions depend on choices and rewards at other states.This motivates planning methods that account for global policy consequences.
- Linear utility: CHVI estimates convex hulls of future rewards for state-action pairs and identifies coverage sets only when utility is linear.It extends Bellman value iteration rather than constituting a MORL method.
- Nonlinear utility: Nonlinear utility can prevent direct dynamic-programming adaptations based on the Bellman equation, motivating specialized planning algorithms.One example minimizes distance between a received reward vector and a target reference point.
7.2 Multi-objective reinforcement learning algorithms
The surveyed MORL algorithms include multi-objective bandits, vector-valued value methods, policy search, Monte Carlo methods, multi-policy strategies, model-based methods, and evolutionary approaches. Their design varies with utility structure, policy representation, observability, and whether preferences are specified before, after, or during learning.
- Stateless/bandit algorithms: Pareto UCB1 outperforms linear and Chebyshev scalarised UCB1 variants, while Pareto knowledge gradient methods achieve further improvements over Pareto UCB1.These comparisons concern multi-objective multi-armed bandit algorithms.
- Value-based methods: Extending Q-learning to MORL requires vector-valued Q-values and a scalarisation function matching the user’s utility for greedy action selection.Linear scalarisation converts the MOMDP into a corresponding MDP, preserving existing convergence proofs.
- Policy-search methods: Policy-search methods can directly optimize nonlinear utility functions and generally produce stochastic policies.These properties can be beneficial for multi-objective problems.
- Nonlinear utility: Under ESR, EUPG uses Monte Carlo simulations over full-episode returns, while DMCTS learns a posterior distribution over their utility.Both methods address the failure of additive Bellman returns under nonlinear utility.
- Multi-policy methods: Outer-loop methods solve a sequence of single-objective problems, whereas inner-loop methods directly produce multiple policies in parallel.Outer-loop efficiency can improve through information reuse and more efficient utility-parameter searches.
- Multi-policy methods: Multi-objective MCTS selects branches using hypervolume or Pareto-dominance measures, while model-based methods can derive policies for new utility functions after learning an environment model.Evolutionary populations likewise represent multiple policies suited to different utility preferences.
- Application settings and preferences: Policy-search extensions have been proposed for continuous state-action spaces and partial observability, and interactive methods allow preferences to be specified during learning.Other work learns user utility from interactions or uses inverse reinforcement learning for preference elicitation.
8 Evaluating the performance of multi-objective decision making algorithms
Multi-objective algorithms often produce solution sets rather than one optimum, making evaluation and comparison more complex. The paper therefore reviews metrics based on Pareto-front properties and user utility.
- Unlike single-objective reinforcement learning, multi-objective algorithms often produce solution sets containing multiple optimal trade-offs.
- Evaluating solution sets requires assessing properties such as spread, coverage, and distance, while determining when one set is better than another.
- The paper contrasts axiomatic metrics with utility-based metrics for evaluating multi-objective algorithms.
- Axiomatic metrics often assume the true Pareto front or convex hull and can be difficult to interpret from a user perspective.
8.1 Axiomatic-based evaluation metrics
Axiomatic metrics evaluate approximate solution sets through properties such as hypervolume, spread, sparsity, approximation distance, and coverage. The paper highlights that these measures can depend on arbitrary references or parameters and may not reflect user utility.
- Hypervolume: Hypervolume measures the value-space volume dominated by an approximate coverage set relative to a reference point.For a two-objective maximisation problem, the reference point defines the region whose dominated volume is measured.
- Hypervolume: The hypervolume reference point may be chosen arbitrarily, affecting the resulting evaluation.
- Hypervolume: Hypervolume is difficult to interpret as user value: extreme solutions can increase it substantially, while valuable nearby solutions may add little.
- Hypervolume: Hypervolume provides limited information: maximising it generally guarantees recovery of a Pareto Coverage Set, but this is not informative during learning.
- Sparsity: Sparsity evaluates Pareto-front approximations by penalising uneven coverage, with lower sparsity considered better.The metric uses sorted objective values across the approximate front.
- ε metric: The ε metric indicates how closely a solution set approximates the Pareto front across all objectives and can compare arbitrary solution sets.The additive form bounds an objective-wise shortfall, whereas the multiplicative form allows each objective to be worse by a factor of 1 + ε.
- ε metric: The paper argues that ε is more useful than hypervolume because it can directly support deriving utility for a user.
- Coverage Ratio: Coverage Ratio combines precision and recall through their harmonic mean when comparing policies against a finite ground-truth Coverage Set.
8.2 Utility-based evaluation metrics
Utility-based metrics evaluate solution sets according to the utility they provide to users rather than only their geometric or coverage properties. The paper presents expected utility and maximal utility loss as alternatives when direct utility assessment is unavailable.
- Utility-based evaluation is preferable when algorithms should be assessed according to the utility their solutions offer users.
- When measurable, user utility can be evaluated directly through outcomes such as revenue achieved by a deployed solution.
- Expected utility: Expected utility measures the utility expected from an algorithm’s solution set under a prior distribution over user utility functions.It is useful when solutions serve different users or are used over time under varying utilities.
- Maximal utility loss: Maximal utility loss measures the greatest utility loss from selecting a policy in a given solution set instead of the full set of potentially optimal solutions.A reference set can be used when computing the full optimal set is infeasible.
- Maximal utility loss: Maximal utility loss is bounded when the solution set has an ε-accuracy bound and the utility function is guaranteed to be Lipschitz-continuous.
8.3 A word of caution regarding value vector approximation
Multi-objective reinforcement learning value vectors are stochastic estimates rather than exact quantities, especially when function approximators are used. The paper recommends independently evaluating candidate policies before selection to reduce utility losses from estimation errors.
- In multi-objective planning, exact policy value vectors can support utility derivation during selection, unlike estimated MORL value vectors.
- MORL value-vector estimates may have high variance or systematic bias, and function approximators can add further variance or bias.
- Inexact estimates can cause both the best policy to be missed and the selected policy to deliver different utility than expected.
- The paper recommends extracting coverage-set policies and conducting separate thorough policy evaluation before selecting a policy for execution.
8.4 Benchmark problems for multi-objective decision making
MORL evaluation depends on benchmarks that support fair comparisons and reflect realistic complexity. Existing benchmarks cover several extensions, but many-objective settings beyond MOMDPs remain poorly represented.
- Why benchmarks matter: Common benchmarks enable fair comparisons because small experiment-design variations can substantially affect reinforcement-learning results.Evaluating several benchmarks also helps assess whether findings generalize.
- Benchmark coverage: Table 2 surveys MORL benchmarks spanning discrete actions, high-dimensional states, partial observability, multiple agents, and continuous actions.The table includes both frequently used benchmarks and newer extensions.
- Beyond MOMDPs: Benchmarks for multi-objective coordination graphs and normal-form games remain few and distributed across separate papers.This limits the availability of common evaluation settings for these problem classes.
9 An illustrated example
The illustrated water-management example uses MONES to discover diverse policies for conflicting supply and flooding objectives without assuming a specific utility function. Compared with repeated single-objective NES runs, MONES produces a more evenly spread solution set and converges within the demonstrated training process.
- Problem setting: The dam must meet downstream water demand while limiting upstream flooding, creating conflicting objectives in a continuous state-action environment.Reservoir inflow and release decisions determine the trade-off between storing water for demand and avoiding floods.
- Problem setting: With unknown, potentially nonlinear utility, the example approximates the Pareto front without assuming decision-maker preferences.The goal is to present alternatives rather than commit to one scalar utility function.
- MONES approach: MONES samples Gaussian-distributed policy parameters, evaluates policies with non-dominance ranking and crowding distance, and updates the distributions using a natural gradient.The indicator combines frontier quality with diversity, encouraging policies across the Pareto front.
- MONES results: After 30 iterations, 11 sampled solutions are non-dominated and 48/50 policies lie reasonably close to the frontier.The sampled policies provide diverse combinations of returns while retaining high solution quality.
- Comparison with NES: NES produces concentrated returns near (−1.0, −10.7), whereas MONES yields a more spread-out coverage set and requires one training run instead of 30.NES samples utility functions and trains a separate single-objective run for each, while MONES directly optimizes the diversity-aware indicator.
- Evaluation metrics: MONES converges toward the approximated optimal solution after 15 iterations and remains stable thereafter, while hypervolume does not fully reflect user utility.At iteration 7, expected user utility nearly doubled but hypervolume increased by only 30%.
10 Conclusion, challenges and open questions
Multi-objective planning and reinforcement learning address real-world tasks with conflicting objectives, but the field remains comparatively niche and faces substantial data, benchmark, scalability, and multi-agent challenges. The paper therefore advocates broader adoption alongside better shared resources and methods that can adapt as objectives change.
- Motivation and scope: Real-world sequential decisions frequently require trade-offs among conflicting objectives, contrasting with the inherently single-objective framing of many planning and RL methods.This mismatch motivates explicitly multi-objective agents for complex applications.
- Data challenges: Insufficient, heterogeneous, unavailable, or weakly correlated data can make some multi-objective problems difficult or impossible to study.These data challenges are especially significant in basic research.
- Benchmark gaps: Existing MORL benchmarks are often simple and do not capture real-world complexity such as many objectives, partial observability, multiple agents, long horizons, and complex state-action spaces.The paper suggests adapting existing single-objective benchmarks by making their reward functions multi-objective.
- Multi-agent challenges: Multi-objective multi-agent systems remain understudied because they combine multiple actors and objectives with increasingly complex dimensions.The literature has also lacked a uniformly accepted framework for comparing contributions and identifying gaps.
- Open questions: MOMADM research still faces open questions about negotiation, equilibria, preference learning, sequential or continuous settings, and evaluation with scarce datasets and benchmarks.These issues make systematic evaluation particularly challenging in multi-agent settings.
- Dynamic objectives: Adding objectives during problem refinement is distinctive to multi-objective methods, motivating approaches that reuse prior experience while minimizing adjustment regret.An experience archive could support offline learning for updated objective specifications without further environment interaction.
- Closing perspective: The article identifies design factors, reviews how current algorithms relate to them, and illustrates applications while encouraging wider adoption of multi-objective agent technologies.Its stated aim is to support future growth of real-world applications based on multi-objective agents.