Source-linked AI summary
It's a matter of timescale: non-linear utility in successor features and multi-objective planning and learning
Liam P. H. Mertens, Lucas N. Alegre, Florent Delgrange, Diederik M. Roijers, Ann Nowé, Peter Vamplew
TL;DR
Existing multi-objective planning and reinforcement learning criteria address non-linear utility at separate timescales, leaving simultaneous effects within one decision problem insufficiently addressed. The paper motivates a multiple-timescale utility perspective through intuition and a toxicity example, finding that different timescale objectives can produce different policies and outcome distributions. It concludes that specialised methods are needed for combined-timescale utility optimisation.
Problem
SER, ESR, and successor-feature approaches address non-linear utility at different timescales, but the paper identifies a gap when multiple timescales occur within one decision problem.
Method
The paper develops a multiple-timescale utility perspective and examines it intuitively and numerically using a toxicity example.
Results
Different policies and outcome distributions arise when utility effects across multiple timescales are optimised simultaneously rather than individually.
Takeaways & Limitations
The results call for comprehensive methodologies and specialised algorithms that optimise combined utility effects across different timescales.
Takeaways & Limitations
The example omits sub-chronic toxicity, which would introduce an additional timescale between SER and ESR.
Abstract
from arXiv · showhide
Time is of the essence when dealing with multiple reward signals and non-linear utility. In this paper we argue that the current main approaches in multi-objectiveRL (SER and ESR), and successor features, are insufficient. While each approach deals with non-linear effects on user utility on different timescales, none of them take into account that different effects happening on different timescales can happen within the same decision problem. We motivate that this can indeed be the case by an example, both intuitively and numerically, leading to a new perspective, and a significant and non-trivial gap in the literature.
1 Introduction
Multi-objective planning and reinforcement learning replaces scalar rewards with reward vectors whose components encode trade-offs, then applies utility over accumulated outcomes. SER, ESR, and successor features each address non-linear utility at different timescales, but the paper identifies a gap when multiple timescales coexist in one decision problem.
- Multi-objective planning and reinforcement learning represents conflicting objectives as a reward vector rather than a single additive scalar reward.Each component corresponds to an objective contributing to a trade-off.
- SER applies utility to expected accrued returns, assuming sufficient time for average outcomes to materialise.This perspective suits settings such as repeated video-processing workloads evaluated by average power consumption, throughput, and latency.
- ESR applies utility to each trajectory’s accrued return before taking the expectation, making it appropriate for one-off outcomes such as a patient’s treatment.SER and ESR coincide for linear utility but differ under non-linear utility.
- Successor features evaluate non-linear utility at individual states or transitions before accumulation, then linearly scalarise the resulting features.This represents utility effects occurring on an immediate per-step timescale.
- The paper argues that non-linear utility effects can occur on different timescales within the same decision problem, exposing a gap in existing approaches.It motivates this claim with a toxicity example and a simplified numerical analysis.
2 Background
Multi-objective RL formalizes competing objectives with vector rewards and requires a utility function plus an optimization criterion to resolve trade-offs. SER, ESR, RSR, and successor features apply utility at different points, producing different optimal policies under non-linear utility.
- Multi-objective decision processes: A multi-objective Markov decision process extends an MDP with an m-dimensional reward function, while policies map states to action distributions.The MOMDP tuple includes state and action spaces, transitions, vector rewards, an initial-state distribution, and discounting.
- Utility and optimization: Multi-objective RL needs both a utility function that scalarises reward or return vectors and an optimization criterion specifying when utility is applied.The placement of utility determines whether the objective concerns average outcomes, complete trajectories, or individual rewards.
- SER and ESR: SER applies utility after averaging policy returns, whereas ESR applies utility to each complete trajectory before averaging across trajectories.SER suits repeated use where aggregate population performance matters; ESR captures the utility of an individual realised outcome, including risk or fairness preferences.
- Returns of scalarised rewards: RSR applies utility to each reward vector immediately and then accumulates the resulting scalar rewards over time.This criterion treats each time step as a separate trade-off and can represent penalties for dangerous instantaneous objective combinations.
- Consequences of non-linearity: When utility is non-linear, SER, ESR, and RSR can have different optimal policies; with linear utility, they share the same optimal policy.Thus, the choice of optimization formulation can change the solution set for the same multi-objective problem.
- Successor features: Successor features represent expected accumulated features and are equivalent to MORL under linear utility, while their expressiveness depends heavily on feature design.Learned or high-dimensional representations can broaden the reward functions an SF approach expresses, whereas MORL typically broadens expressiveness through non-linear utility.
- Connecting MORL and SFs: Non-linear RSR can be implemented in successor features by appending non-linear feature combinations, but traditional SFs generally cannot implement SER or ESR.After rewards are accrued, traditional successor-feature approaches allow only linear combinations, limiting their representation of post-accumulation utility.
3 Motivating Example: Toxicity
The toxicity example shows that acute, sub-acute, and chronic effects can coexist in one scheduling problem, requiring utility to be evaluated across multiple timescales simultaneously.
- Acute exposure: Acute toxicity reflects immediate high-dose effects and is modeled by applying a nonlinear response to each task’s daily dose.This corresponds to applying utility per timestep before rewards or penalties are accumulated.
- Chronic exposure: Chronic toxicity concerns long-term damage from small exposures over multiple months or years, corresponding to the SER timescale.The example links this setting to minimizing long-term average dose across employees under a sigmoid response.
- Sub-acute exposure: Sub-acute toxicity depends on an employee’s cumulative dosage over an entire month and requires tracking prior exposure over time.The monthly outcome is treated as a probability distribution whose expected adverse effects are evaluated with a nonlinear response curve.
- Multi-timescale optimization: SER, ESR, and RSR each capture one toxicity timescale, but a responsible schedule must optimize acute, sub-acute, and chronic exposure together.The paper identifies simultaneous multi-timescale optimization as a gap in the literature and proposes combining acute and episodic utility functions.
- Numerical decision problem: The scheduling problem assigns three employees with different skills to three tasks with varying toxicity while limiting toxic dosage across several time horizons.It is modeled as a MOMDP whose states are employee-task permutations and whose vector reward records each employee’s daily toxic dose.
4 Experiments
The experiments compare criteria tailored to ESR, SER, successor features/RSR, and a combined multi-timescale objective on an illustrative toxicity assignment problem. They produce distinct policies, including switching, sacrificing, and skill-matched strategies, showing that combined optimization can expose safety trade-offs missed by individual criteria.
- Experimental setup: The study evaluates ESR, SER, RSR, and a combined objective using tailored algorithms, with MONES serving as a heuristic for the combined utility.The experiments are explicitly presented as an illustrative proof of concept showing differences between optimization criteria.
- Results: ESR learns a switching strategy that keeps toxicity near the threshold of 400 for all employees, while sometimes exceeding daily intake limits.The policy balances toxicity across employees over the episode but does not always satisfy the daily constraint.
- Results: SER rotates employees 1 and 2 below the threshold while sacrificing employee 3, producing higher utility for two employees and increased variance.The expected returns for employees 1 and 2 remain close to ESR, but with greater variance, indicating a risk-sensitive weakness of SER in this example.
- Results: RSR learns a switch-averse policy that assigns employees to tasks matching their skills, thereby avoiding additional total-toxicity requirements.This policy is described as certain to keep working under the example’s conditions.
- Results: The combined method mainly assigns employee 3 to the most difficult task and prioritizes the utilities of the two most experienced employees.The authors suggest that combining utilities over timescales can permit maximizing one or two utilities while neglecting another.
- Interpretation: The example’s combined objective raises a safety issue because not all employees can remain safe from chronic exposure, unlike the balancing policies under RSR or ESR.The authors characterize the setup as too hard for all employees to be safe and state that additional employees or better hazard mitigation would be needed.
5 Related Work
Related work studies how utility criteria, distributional value representations, and successor-feature extensions affect policy expressivity and risk-sensitive evaluation. These lines of work provide components for richer reward representations or dynamic preferences, but the paper focuses on utility effects occurring across multiple timescales.
- Multi-objective criteria: Most MORL algorithms optimize a single criterion, while prior work has examined augmented-state ESR, reduced-variance SER, and the policy-ordering relationships between SER and ESR.The cited literature includes constructions preserving optimal policies under ESR and theoretical comparisons of expressible policy orderings.
- Distributional and successor-feature methods: Distributional successor features enable zero-shot risk-sensitive policy evaluation, treating risk-sensitive evaluation as a form of non-linear utility.The paper identifies this as an example of combining expressive reward representations with non-linear utilities.
- Dynamic preferences: Option Keyboard methods dynamically change the preference vector and associated policy at each timestep instead of fixing a single preference vector.This connects successor-feature-based methods with dynamic preferences in MORL.
6 Discussion
The paper argues for a utility-based perspective that handles multiple timescales simultaneously, supported by intuitive and numerical toxicity examples. It identifies omitted timescales, limited empirical scope, and difficult policy optimization as boundaries for the conclusions.
- Discussion: The paper demonstrates the need to optimize utility across multiple timescales simultaneously through intuitive and numerical toxicity examples.The authors frame this as a utility-based perspective for multi-objective planning and reinforcement learning.
- Scope and omitted effects: The example omits sub-chronic toxicity, which would represent an intermediate timescale and could be added as a fourth timescale using three policy executions.The paper notes uncertainty over the exact sub-chronic boundary and uses approximately 90 days as a possible interpretation.
- Scope and motivation: The paper uses only one abstract example, while motivating broader relevance through customer-order completion times and individual waiting-time concerns.The example contrasts average service cost with unusually long waits that can produce bad reviews and reputational damage.
- Implications: Different policies work better for different timescales, so methods successful for SER, ESR, or RSR may not suffice for their combination.The authors call for comprehensive methodology and algorithms for simultaneous multi-timescale optimization.
- Future methodology: A possible solution requires distributional return and reward functions, but policy optimization remains difficult because policy changes affect multiple timescales.The paper points to complex utility landscapes and mentions policy gradients and evolutionary strategies as possible tools.
Ethical Statement
The toxicity example is purely illustrative, not a recommendation for how organizations should manage employee toxin exposure. More broadly, the paper links multiple rewards and optimization criteria to explicitly incorporating ethical considerations into AI decision-making.
- The toxicity example is illustrative and does not advocate managing employees’ toxin exposure in this manner.
- Multiple rewards and optimization criteria may enhance AI agents’ capacity to include ethical considerations explicitly during decision-making.