Multi-Objective Reinforcement Learning Using Routed Ensembles and Trajectory Attribution

1 Introduction

• Optimal policies may change qualitatively with small preference shifts, and whether trajectories transfer across objectives remains unclear.With aligned objectives, data collected under one preference can benefit others; with misaligned or corrupted objectives, naive transfer can be inefficient or harmful.Source: §1 · pp. 1–2
• The objective conflict coefficient measures objective misalignment through χ²-divergence, and its interaction with coverability determines online MORL sample complexity.The regret bound formalizes how misalignment affects learning efficiency across preferences under a low-rank basis assumption and a condition ensuring sufficient exploration of the basis.Source: §1 · p. 2
• Routed ensemble policy learning maintains basis policies, routes each trajectory by expected return under its preference, and updates each policy using its assigned cluster.The updates can use a standard policy-learning algorithm such as proximal policy optimization.Source: §1 · p. 2
• Kernel-based trajectory attribution fits a kernel ridge regression surrogate across trajectory subsets to estimate how trajectories help or hinder policy learning.The kernel captures individual contributions and higher-order interactions that first-order influence approximations and linear surrogates do not capture.Source: §1 · p. 2
• Routed ensemble learning outperforms the strongest baseline by 4.3% on Meta-World and achieves an 8% average improvement in hypervolume on MO-Gymnasium tasks.Kernel-based attribution improves Spearman correlations with ground-truth leave-one-out influence scores by 50%; kernel-based selection yields 27% higher reward than existing attribution baselines.Source: §1 · p. 2

2 Preliminaries

• A multi-objective Markov decision process has state and action spaces, a transition kernel, a vector-valued reward with at least two objectives, and a horizon.The model also includes a preference space and a scalarization operator that maps return vectors to scalar utility under a preference.Source: §2 · p. 3
• In the episodic online setting, the learner observes a preference each episode, selects a policy using past trajectories and preferences, and aims to learn a preference-conditioned policy for target preferences.The paper focuses on an ensemble corresponding to a small set of policies.Source: §2 · p. 3
• Coverability measures how difficult it is for one exploration distribution to cover the occupancy measures of all policies in a fixed policy class.The passage defines occupancy through state-action probabilities induced by policies.Source: §2 · p. 3

3.1 Examples and motivation

• Aligned objectives share one optimal policy, linear conflict is spanned by corner policies, and concave conflict requires intermediate policies; corrupted objectives motivate identifying trajectories aligned with the target preference.With multiple objectives, different pairs can exhibit different alignment and trade-off regimes.Source: §3.1 · p. 4

3.2 Theoretical characterization through conflict coefficients

• Γ measures the diversity of basis-policy occupancies relative to their mixture, while K represents the intrinsic complexity of objective trade-offs through the number of basis policies.Smaller Γ corresponds to greater occupancy overlap and easier transfer; near-disjoint occupancies correspond to greater learning difficulty.Source: §3.2 · p. 4
• The regret analysis separates exploration hardness, captured by Ccov, from transfer hardness, captured by Γ; a matching lower bound establishes near-tight dependence on Γ and T up to Ccov and logarithmic factors.The same qualitative dependence on K and Γ is preserved when the exact low-rank basis assumption is relaxed to approximate low-rankness.Source: §3.2 · p. 5
• In the tabular concave setting, mean convex-hull residual decreases from 0.186 at K = 3 to 1.45 × 10−4 at K = 21; on MO-Hopper it decreases from 0.443 at K = 3 to 0.231 at K = 15.The MO-Hopper values use candidate-selected bases.Source: §3.2 · p. 5

3.3 Learning routed ensemble policies

• Routed ensemble learning scores trajectories under each basis policy, routes each to the highest-scoring policy, and updates each policy using only its assigned trajectories.When the composite policy lies in the convex hull of fixed basis policies, optimal routing deterministically selects the policy with the highest return under the preference.Source: §3.3 · p. 6

3.4 Trajectory attribution

• The kernel-surrogate method samples trajectory subsets, estimates their performance, and fits kernel ridge regression; its nonlinear model can capture synergy and antagonism among trajectories.Trajectory scores can be formed by averaging over subsets containing each trajectory, then used to select trajectories for downstream applications.Source: §3.4 · p. 7
• Kernel-model predictions correlate positively with leave-one-out scores and negatively with Γ; Algorithm 2 reaches Spearman rank correlation 0.75 with true policy-gradient influence scores.The surrogate estimates subset performance around a reference policy using local policy-gradient rollouts rather than full retraining.Source: §3.4 · p. 8

4 Experiments

• Routed ensembles outperform baselines by a 4.3% relative margin in multi-task RL, up to 48% in multi-objective RL, and 17% in large language model benchmarks; kernel surrogates improve correlation by 50% and task-selection reward by 27%.Source: §4 · p. 8

4.1 Evaluation in synthetic environments

• Routed ensembles substantially reduce cumulative regret on four-objective Pareto-frontier learning and mitigate negative transfer from corrupted examples.The synthetic evaluation also varies goal separation and measures Γ from the resulting occupancies against routing’s regret reduction.Source: §4.1 · pp. 8–9, Figure 2 · p. 9

4.2 Evaluation on robotic control and language models

• On MT10, routed ensembles achieve 85.6% average success rate versus soft modularization’s 82.0%, a 4.3% relative margin.Source: §4.2 · p. 9, Table 1 · p. 10
• Routed ensembles improve hypervolume over single-policy and uniform-assignment baselines on RLHF benchmarks and yield an 8% average improvement over Q-Pensieve in continuous MO-Gymnasium.On HelpSteer2, improvements average 5.0% and 8.2% over the single-policy and uniform baselines; on UltraFeedback, they are 6.2% and 3.0%. The continuous-task comparison also reports higher HV than learned-gate MoE and GPI-LS on all three benchmarks.Source: §4.2 · p. 10, Table 2 · p. 10, Table 3 · p. 11
• Kernel-surrogate attribution improves LDS by 50% over attribution baselines and raises target reward by 27% when selecting source tasks for co-training.Source: §4.2 · p. 10, Table 5 · p. 11

4.3 Ablation studies

• Routed ensembles reduce the synthetic environment’s measured coverability χ2-divergence from 1.3 for a single policy to 0.2.Across increasing goal separation, Γ rises from 0 to 1 while routing’s regret reduction rises from 0 to 36.03.Source: §4.3 · pp. 10–11, Table 4 · p. 11
• The highest reported hypervolumes occur at N = 3 for MO-HalfCheetah-v4 and MO-Hopper, and N = 5 for MO-Ant.The reported values are 5.21 × 10^6, 1.87 × 10^7, and 2.12 × 10^7, respectively.Source: §4.3 · p. 10
• Quadratic, cubic, and RBF kernels each reach fitting loss 0.1, versus 0.2 for the linear kernel, indicating higher-order interactions in objective subsets and policy returns.Source: §4.3 · p. 11

• MORL approaches either scalarize objectives to learn a single policy for a fixed or sampled preference, or learn multiple policies to address sharp variation in optimal policies across preferences.Source: §5 · p. 11
• Recent work partitions objectives into disjoint related groups through a convex relaxation; this paper extends that line with a theoretical characterization of conflict, routed ensembles, and trajectory-level attribution.Source: §5 · p. 11
• Influence estimation has moved from supervised learning into RL, while online RL attribution faces circular dependence between policy updates and future data distributions.Prior approaches include offline attribution to policy decisions and online local data attribution using gradient similarity to identify and filter harmful transitions.Source: §5 · p. 11

6 Conclusion

• The paper combines a routed ensemble guided by objective conflict with kernel-based trajectory attribution to address competing objectives.The attribution method captures nonlinear interactions among trajectories and can improve subsequent policy learning.Source: §6 · p. 12
• The theoretical regret bounds do not directly apply to the routed ensemble algorithm.Understanding the sample complexity of ensemble policies in multi-objective Markov decision processes remains future work.Source: §6 · p. 12

A Policy Gradient Estimation

• The kernel surrogate fits a kernel ridge regression model to estimated subset performance, representing nonlinear interactions among trajectory subsets.It forms the kernel matrix Ψ, uses estimated values v(Si), and computes coefficients α = (Ψ + λ Id)−1V.Source: A Policy Gradient Estimation · p. 15
• A first-order PPO surrogate is converted into weighted binary classification to obtain a bounded estimate of policy performance for trajectory subsets.The labels are the signs of estimated advantages and the sample weights are their absolute values.Source: A Policy Gradient Estimation · p. 15
• Approximation error stays below 2% for small ∥θ − θ∗∥ with proper initialization and rises to around 10% for perturbations up to 5%.The results are averaged over 10 randomly sampled subsets across MT10, CartPole, LunarLander, and Highway.Source: A Policy Gradient Estimation · p. 15, Table 6 · p. 15
• Random Gaussian projections reduce high-dimensional gradients while preserving pairwise similarity, and the reduced-space regression takes only a few seconds.Source: A Policy Gradient Estimation · pp. 15–16
• The attribution evaluation compares predicted subset rankings with true downstream effects using Spearman rank correlation, and reports strong correlation with leave-one-out results.Source: A Policy Gradient Estimation · p. 16
• The approach naturally extends to online multitask RL, but further evaluation on those benchmarks is left for future work.Source: A Policy Gradient Estimation · p. 17

B Complete Proofs

• The appendix outlines a complete proof of the theorem, a matching lower bound, analysis of the conflict coefficient, and a relaxation to an approximate low-rank basis.Source: B Complete Proofs · p. 17

B.1 Proof of Theorem 3.6

• The proof reduces cumulative regret to on-policy Bellman residuals, split into an occupancy-conflict factor and an in-sample residual factor.The conflict factor is controlled by Assumption 3.5; the in-sample factor uses witness comparison and pooled regression guarantees.Source: §B.1 · pp. 17, 19
• After burn-in, the stable-phase bound depends on the conflict coefficient, and the burn-in contribution is O(Ht0), with t0 = O(CcovKβ).The proof obtains the stated worst-case scaling using 1 + Γ ≤ K.Source: §B.1 · pp. 17, 20
• Pooled confidence updates use cumulative occupancy across all previous episodes, rather than only trajectories routed to the selected basis policy.Source: §B.1 · pp. 17, 20
• For any basis, the conflict coefficient is at most K − 1, with equality up to lower-order terms when basis occupancies have disjoint supports.Source: §B.1 · pp. 20–21

B.2 Proof of the regret lower bound

• Theorem B.6 gives a regret lower bound of Ω(H (1 + Γ)T) for a family with K basis policies, Ccov ≤ K, and Γ = K − 1.Source: §B.2 · pp. 21–22
• The upper and lower bounds match in their dependence on K and Γ up to factors of Ccov log |F|.Source: §B.2 · p. 21
• The lower bound embeds one Bernoulli two-armed bandit in each of K pairwise-disjoint sub-chains, with a cyclic preference sequence.Regret comes from mis-routing or choosing the wrong binary action within a correctly routed sub-chain.Source: §B.2 · p. 21

B.3 Bounds on objective conflict coefficients

• For the routed-ensemble replay mixture, the off-policy estimator variance is inflated by exactly 1 + Γ(t).Source: §B.3 · p. 22

The importance-weighted off-policy estimator

• The importance-weighted off-policy estimator is unbiased under the stated replay-sampling setup.Source: §B.3 · p. 23, The importance-weighted off-policy… · p. 23

B.4 Approximate low-rank conditions

• Approximate decomposition inflates the conflict factor by ε, and true regret additionally includes the gap between the true optimum and the best basis policy.The basis-relative bound remains sublinear; when ε = 0 and the basis is exact, basis-relative and true regret coincide.Source: §B.4 · pp. 23–24
• Assumption B.9 requires each deployed policy to use one mixing vector across stages, with occupancy error bounded by ε.The proof’s norm uses a support convention: finite norm requires zero error wherever the reference occupancy is zero.Source: §B.4 · p. 23
• Stable coverage after burn-in and Bellman completeness still support pooled regression without exact basis decomposition.Coverage may be assumed directly or derived from approximate exploration that makes accumulated occupancies dominate the basis mixture.Source: §B.4 · pp. 24–25
• The burn-in contribution is bounded by Ht0, with t0 = O(CcovKβ), before the post-burn-in regret bound applies.The cumulative basis-approximation error is at most HεT, separating approximation cost from basis-relative regret.Source: §B.4 · pp. 25–26

C Omitted Experiments and Implementation Details

• MO-Gymnasium experiments take approximately 72 hours; Meta-World takes 12 hours and RLHF takes 1 hour on the stated server setup.The appendix names an AMD Ryzen 7 9700X CPU for MO-Gymnasium and an Intel Xeon E5-2623 v3 CPU with NVIDIA RTX A6000 GPUs for Meta-World and RLHF.Source: C Omitted Experiments and… · p. 26
• Kernel attribution predicts a conflicting objective’s marginal gain by comparing surrogate performance with and without that objective.The surrogate is kernel ridge regression trained on trajectory-subset masks and their outcomes.Source: C Omitted Experiments and… · p. 27
• Routed ensembles use Soft Modularization for Meta-World, Q-Pensieve for continuous MO-Gymnasium, and PQL for discrete MO-Gymnasium.Baselines use official implementation hyperparameters; training runs for 5 × 10^5 steps on continuous and 10^4 on discrete benchmarks.Source: C Omitted Experiments and… · p. 27
• The attribution implementation uses an RBF kernel with γ = 0.1 and regularization λ = 0.1, selected by validation-set cross-validation.Source: C Omitted Experiments and… · p. 27

C.1 Synthetic environments

• The two-objective parametric task includes aligned goals, a barrier-induced linear frontier, and a concave frontier where intermediate policies outperform corner-solution mixtures.The aligned goals are proximal; the conflicting cases separate the targets, with the barrier removed for the concave-frontier case.Source: §C.1 · p. 27
• A separate synthetic environment creates a synergy trap in which two objectives are individually neutral or detrimental but beneficial when combined.Source: §C.1 · p. 27
• The grid-world MOMDP uses distinct goal states and negative Manhattan-distance rewards, then computes preference-optimal policies and occupancies for conflict measurement.The conflict coefficient compares a candidate occupancy with the uniform average of single-objective expert occupancies.Source: §C.1 · p. 28
• A single scalarized policy has high χ²-divergence from the target occupancy mixture, while a routed ensemble with K basis policies significantly reduces it.The tabular setup permits exact occupancy and divergence calculations, avoiding deep-RL approximation errors.Source: §C.1 · p. 28

C.2 Additional empirical results

• The mean occupancy residual in the concave tabular setting falls from 0.186 with three bases to 1.45 × 10−4 with 21; on MO-Hopper, 15 candidate-selected policies yield 0.231 versus 1.33 for three learned policies.The measurements show larger candidate bases can fit evaluated occupancies more closely, but the learned ensemble need not recover the assumed structure.Source: §C.2 · p. 29, Table 8 · p. 29
• With 21 preference-optimal policies but only two learned policies over 3000 episodes, GridWorld yields mean and maximum occupancy residuals of 0.475 and 3.474 and cumulative value excess of 2780.4.Source: §C.2 · p. 29
• RLHF evaluations cover synthetic helpfulness-safety data and four model–dataset configurations using Qwen3-1.7B or Llama-3.2-1B with HelpSteer2 or UltraFeedback.All configurations train per-basis LoRA adapters using PPO.Source: §C.2 · p. 29
• Figure 4 compares ensemble-size choices across MO-HalfCheetah2d, MO-Hopper, and MO-Ant using hypervolume.Source: §C.2 · p. 29, Figure 4 · p. 30

C.3 Proximal policy optimization

• PPO collects rollout records containing transitions and auxiliary values, then updates policy parameters by maximizing a clipped surrogate objective; the theory treats advantage estimates as given.Generalized advantage estimation is one possible estimator, but the analysis is independent of the particular estimator.Source: §C.3 · p. 30