Source-linked AI summary
Preference-Guided Cooperation in Multi-Objective Multi-Agent Reinforcement Learning
Qiyue Xia, J. Michael Herrmann
TL;DR
MOMARL combines agent coordination with trade-offs among potentially conflicting objectives, but existing approaches have largely treated objective structure as an added optimisation dimension. This paper introduces preference-guided MIRL and reports improved Pareto-front quality over baselines, with MIRL-D performing strongest overall.
Problem
Existing MOMARL approaches have largely treated multiple objectives as an additional optimisation dimension rather than as structure that can guide multi-agent learning.
Source: §1 · p. 1Method
MIRL combines team-level and learnable individual preferences with objective-wise counterfactual credit assignment and meta-gradient preference adaptation.
Source: §1 · p. 1Results
MIRL variants consistently improve Pareto-front quality over baselines, with MIRL-D achieving the strongest overall performance.
Source: §1 · p. 2Takeaways & Limitations
The paper’s supported conclusion is that multi-objective structure can be exploited to facilitate multi-agent cooperation while preserving the system-level objective trade-off.
Source: §6 · p. 9Individual preferences are state-independent because each agent uses a single meta-parameter vector throughout interaction.
Source: §4.3.2 · p. 6
Abstract
from arXiv · showhide
Multi-objective multi-agent reinforcement learning (MOMARL) addresses agent coordination and objective trade-offs simultaneously while optimising overall system performance. Existing work tends to treat multi-agent cooperation and multi-objective optimisation as largely independent components of a learning problem, which amounts to an exorbitant complexity except in trivial cases. We study here whether the structure provided by multiple objectives can itself be exploited to facilitate cooperation, rather than being treated as an additional optimisation challenge. We formulate MOMARL at both the team and individual levels and introduce objective-wise counterfactual credit assignment to evaluate each agent's contribution across the team objectives. We propose Meta-Gradient Informed Multi-Objective Multi-Agent Reinforcement Learning (MIRL), with static (MIRL-S) and dynamic (MIRL-D) variants. By incorporating meta-gradient information, MIRL enables agents to learn how to prioritise objectives individually while collectively optimising the system-level objective, allowing the multi-objective structure to directly guide cooperation. Experiments on a multi-objective multiple particle environment show that our methods improve Pareto-front quality over baselines, with the dynamic variant achieving the strongest performance.
1 Introduction
The paper reframes multi-objective structure as a resource for cooperation and introduces adaptive preferences and objective-wise credit assignment within MIRL. Its static and dynamic variants improve Pareto-front quality over baselines, with MIRL-D strongest overall.
1 Introduction
- MOMARL approaches often treat multiple objectives as an added optimisation dimension rather than structure that can guide multi-agent learning.The paper instead argues that objective structure can facilitate cooperation.Source: §1 · p. 1
- A team preference specifies the system-level trade-off, while learnable individual preferences adaptively guide each agent’s behaviour.The preferences form a bi-level formulation.Source: §1 · p. 1
- Objective-wise counterfactual credit assignment evaluates each agent’s contribution separately across team objectives and supplies objective-specific learning signals.Source: §1 · p. 1
- Meta-gradient updates treat individual preferences as meta-variables shaped by their effects on subsequent team performance; MIRL-D additionally adapts them to cooperative context.Source: §1 · pp. 1–2
- MIRL variants consistently improve Pareto-front quality over baselines, with MIRL-D achieving the strongest overall performance.Experiments are conducted in the Multi-Agent Particle Environment.Source: §1 · p. 2
2 Background
The background frames cooperative MARL as decentralized learning under shared rewards, then extends that setting to multiple objectives and Pareto trade-offs.
2.1 Cooperative Multi-Agent Reinforcement Learning
- In cooperative MARL, agents use local observations and individual policies to pursue high shared team performance despite coordination challenges.The setting is formalized as a Dec-POMDP with a shared team reward.Source: §2.1 · p. 2
2.2 Multi-Objective Multi-Agent Reinforcement Learning
- MOMARL extends the cooperative setting to a shared vector-valued reward, with preferences specifying trade-offs among team objectives.Different preferences can lead to different optimal joint policies.Source: §2.2 · p. 2
2.3 Pareto Optimality in MOMARL
- A joint policy is Pareto optimal when no other policy dominates it, and practical methods seek finite policy sets approximating the Pareto front.The front represents expected vector returns of Pareto-optimal policies.Source: §2.3 · pp. 2–3
3 Related Work
Prior work addresses MORL, cooperative MARL, and MOMARL through distinct methods, but comparatively little work uses objective structure to shape individual cooperation.
3 Related Work
- MORL methods either scalarize objectives into a single-policy task or learn multiple policies to approximate diverse Pareto trade-offs.Multi-policy approaches include preference-conditioned models, generalized policy improvement, and successor-feature representations.Source: §3 · p. 3
- Cooperative MARL research addresses non-stationarity and credit assignment through CTDE, value decomposition, counterfactual baselines, communication, and role specialization.Shared rewards can obscure individual contributions and permit poorly contributing agents to receive high team rewards.Source: §3 · p. 3
- MOMARL work spans scalarized applications, credit-assignment methods, utility analyses, deep methods, benchmarks, continuous spaces, and heterogeneous utilities.Examples include MO-MIX, MOMAland, MOSMAC, and MOMA-AC.Source: §3 · p. 3
- Earlier MOMARL work emphasizes system-level optimisation, while this paper adds an individual-level learning-to-learn perspective to adapt agent contributions.The paper treats multiple objectives as potential structure for facilitating cooperation, not only as an added optimisation difficulty.Source: §3 · p. 3
4 Methods
MIRL separates the system’s desired objective trade-off from agents’ learnable preferences, then uses objective-wise counterfactual credit and meta-gradients to guide their contributions. MIRL-D replaces static preferences with context-dependent ones.
4.1 Team-Level and Individual-Level Modelling
- The team preference specifies the system-level trade-off, while learnable individual preferences let agents flexibly emphasise the same objectives without changing team-level evaluation.The relationships among individual preferences emerge through learning rather than being predefined.Source: §4.1 · pp. 3–4, Figure 1 · p. 4
4.2 Objective-Wise Counterfactual Credit Assignment
- Objective-wise counterfactual credit represents each agent’s contribution as a vector across team objectives, rather than reducing it immediately to one scalar.The counterfactual baseline varies agent i’s action while holding other agents’ actions fixed; the method uses centralised training with decentralised execution.Source: §4.2 · p. 4
4.3.1 Preference-Guided Actor Update.
- In MIRL-S, each agent’s individual preference combines its objective-wise counterfactual credit into the scalar learning signal used to update the actor.Individual losses are averaged to update shared actor parameters; the centralised vector critic is trained independently of the preference meta-parameter.Source: §4.3.1 · p. 5, §4.3 · p. 5
4.3.2 Meta-Gradient Preference Update.
- MIRL-S updates each agent’s preference meta-parameter according to how its induced actor update affects a subsequent team-level meta-objective evaluated under the fixed team preference.The method approximates accumulated sensitivity with a meta-gradient trace; experiments use μ = 0, considering only the current meta-parameter’s effect on the immediately resulting actor update.Source: §4.3.2 · p. 6
- Because MIRL-S uses one meta-parameter vector per agent, its learned preference is state-independent and cannot respond to the current observation, behaviour, or cooperative situation.MIRL-D addresses this boundary by generating preferences from the agent’s current local representation.Source: §4.3.2 · pp. 6–7, §4.4 · p. 7
4.4 Dynamic Meta-Gradient Informed Preference Adaptation
- MIRL-D generates each agent’s preference from its current local representation, allowing preferences to vary across agents and timesteps; its critic and objective-wise credit assignment remain as in MIRL-S.The generated preference is supplied to the actor and scalarises objective-wise credit during policy learning.Source: §4.4 · p. 7
5 Experiments
Across four multi-objective particle-environment tasks, MIRL-D achieves the strongest overall Pareto-front performance. The experiments also show that the value of preference adaptation varies with task coordination demands.
5.1 Experiment Setup
- The evaluation extends MPE tasks to two objectives and tests Spread, Formation, Line, and Reference using HV, EU, and SP.The added objective captures energy use; Reference involves two agents and requires communication for successful cooperation.Source: §5.1 · p. 7
5.2 Results
- DTDE-PPO is the weakest PPO baseline, while centralized training improves Pareto-front quality and coverage; CTDE-PPO-Q generally outperforms CTDE-PPO-V.The authors suggest joint-action conditioning gives the critic a more informative coordination signal.Source: §5.2 · pp. 7–8
- MIRL variants consistently achieve strong Pareto-front quality across the benchmarks, with MIRL-D showing the best overall performance.Figure 4 compares the estimated fronts; Table 1 reports quantitative results as mean ± standard deviation.Source: §5.2 · p. 7, Figure 4 · p. 8
- MO-MIX produces dense solutions with low SP but covers a more limited objective-space region, yielding substantially lower HV and EU.The authors suggest a shared preference-conditioned model and discretization of continuous actions may constrain performance.Source: §5.2 · p. 8
- MO-COMA outperforms PPO baselines across tasks, approaches MIRL on Formation and Line, and trails MIRL more clearly on Spread and Reference.Source: §5.2 · p. 8
5.3 Ablation and Further Analysis
- On Spread, MIRL-S improves over MO-COMA and MIRL-D adds further gains; on Reference both improve but dynamic adds little, while Formation and Line leave less room.The authors relate these differences to changing versus relatively stable coordination requirements.Source: §5.3 · p. 8
6 Conclusion
The conclusion presents MIRL as a way to use multi-objective structure to guide cooperation through team- and individual-level preferences, objective-wise credit assignment, and meta-gradient-informed adaptation. The appendix describes how its variants allocate training and retain policies.
6 Conclusion
- MIRL combines team- and individual-level preferences, objective-wise counterfactual credit assignment, and meta-gradient-informed adaptation; MIRL-D achieves the strongest overall benchmark results.The conclusion reports added benefits from preference adaptation particularly where cooperation requires greater adaptability.Source: §6 · p. 9
A Algorithm
- Training is divided equally across M fixed team preferences, with one independently trained policy for each preference.Source: A Algorithm · p. 10
- MIRL-S learns static agent preferences, while MIRL-D uses a shared controller to generate dynamic preferences; only non-dominated policies are retained.Agent preferences are initialized to approximate the team preference.Source: A Algorithm · p. 10
B.1 Benchmarks
The benchmark suite extends particle-environment tasks to two objectives: task reward and energy use. It covers spatial coordination tasks and a communication-dependent reference task.
B.1 Benchmarks
- Spread, Formation, and Line each use three agents and require different spatial arrangements; Reference uses two agents and an explicit communication channel.The tasks derive from MPE, where agents and landmarks move in a two-dimensional continuous space.Source: §B.1 · pp. 10–12
- Each agent’s reward vector includes the underlying task reward and energy consumed; both objectives are maximized to encourage task completion and energy efficiency.Source: §B.1 · pp. 10–11
- In Reference, agents must communicate private target information while moving toward their assigned landmarks, and communication energy depends on transmitted packet length.Silence incurs no communication cost.Source: §B.1 · p. 12
B.2 Evaluation Metrics
The section defines three metrics for evaluating approximate Pareto fronts: hypervolume, expected utility, and sparsity. They assess front quality, average performance across preferences, and solution distribution, respectively.
B.2 Evaluation Metrics
- Hypervolume measures the region dominated by an approximate Pareto front relative to a reference point dominated by all its solutions.A higher hypervolume indicates a front closer to and more extensive with respect to the true Pareto front.Source: §B.2 · p. 12
- Expected utility evaluates the average performance of the policy set over preferences, with higher values denoting better average performance.Source: §B.2 · p. 12
- Sparsity measures gaps between sorted objective values in an approximate Pareto front, with lower values indicating a more uniform solution distribution.For a singleton solution set, sparsity is reported as “–” because its zero value does not meaningfully measure distribution.Source: §B.2 · p. 12
B.3 Training Details
The training details specify a shared training budget and preference scheme, then describe the baseline and proposed method implementations. They also report the computational setup used for the experiments.
B.3 Training Details
- Each environment and seed uses 5 × 10^6 joint steps, with 3 seeds, horizon 25, discount 0.99, and 21 uniformly spaced team preferences for all methods except MO-MIX.MO-MIX trains one preference-conditioned network with the full budget and samples preferences at the same 0.05 interval.Source: §B.3 · p. 12
- The PPO baselines progress from fully decentralised DTDE-PPO to centralised state-value CTDE-PPO-V and joint-action CTDE-PPO-Q critics, while retaining decentralised execution.Their hyperparameters are listed in Table 2.Source: §B.3 · p. 12, Table 2 · p. 13
- MO-COMA extends COMA [9] to vector-valued rewards, and MIRL-S and MIRL-D add static and dynamic meta-gradient preference adaptation to the same implementation.The methods’ hyperparameters are listed in Table 3.Source: §B.3 · pp. 12–13, Table 3 · p. 13
- MO-MIX follows, discretises movement into five actions, and uses 55 discrete actions per agent in Reference by combining movement with 11 communication choices.It is trained as a single preference-conditioned policy; its hyperparameters are listed in Table 4.Source: §B.3 · p. 13, Table 4 · p. 13
B.4 Computational Resources
- Experiments ran on a workstation with an NVIDIA GeForce RTX 5090 GPU, Intel Core Ultra 7 270K CPU, and 64 GB memory, using one GPU per run.Source: §B.4 · p. 13
C Additional Results
Additional results show that MIRL learns agent-specific preferences during training and that both variants adapt those preferences away from their common initialisation. During execution, MIRL-D generates preferences that vary across agents and timesteps despite a fixed team preference.
C Additional Results
- On Line, MIRL-S preferences initialized near the team preference diverge across agents during training, indicating learned agent-specific objective emphases.Figure 5 shows two representative runs under different team preferences.Source: C Additional Results · p. 13, Figure 5 · p. 13
- On Spread, MIRL-S preferences gradually separate during training, while MIRL-D shows a larger overall shift and greater short-term variation.Figure 6 compares a representative run for both variants under team preference (0.35, 0.65); MIRL-D values are batch means.Source: C Additional Results · pp. 13–14, Figure 6 · p. 13
- On Spread evaluation episodes with fixed team preference (0.35, 0.65), MIRL-D preferences vary across agents and timesteps, and trajectories differ between episodes.No parameter updates occur during evaluation; Figure 7 shows two example episodes.Source: C Additional Results · p. 14, Figure 7 · p. 14
D Limitations
The discussion identifies limits on interpreting learned preferences as agent roles and on the breadth of the current evaluation.
D Limitations
- The benchmark tasks do not cleanly separate behavioural tendencies or cooperative roles, limiting quantitative links between preference changes and specific agent behaviours.A systematic link would require environments where those tendencies can be independently identified and measured.Source: D Limitations · p. 14
- The evaluation covers only two-objective cooperative tasks with relatively small agent populations, leaving scalability and generality in larger systems and objective spaces unassessed.Source: D Limitations · p. 14