Pareto Manifold Learning: Tackling multiple tasks via ensembles of single-task models

1. Introduction

• Competing objectives mean that improving one task can come at another’s expense, so practitioners must balance tradeoffs rather than expect one optimal solution.The introduction notes that a single solution satisfying all objectives optimally rarely exists.Source: §1 · p. 1
• Pareto Manifold Learning interpolates among single-task predictors in weight space and weights task losses in tandem with each predictor’s participation.The resulting weight subspace explicitly encodes tradeoffs and continuously parameterizes the Pareto front.Source: §1 · p. 1
• The geometric premise is that nearby Pareto stationary points can be connected by paths whose parameterization maps monotonically in objective space.The authors assume acceptable solutions lie in the intersection of low-loss valleys across tasks.Source: §1 · pp. 1–2
• A single training run produces a subspace of Pareto-optimal solutions that practitioners can select from during inference to suit their tradeoff needs.The authors also link enforcing low loss across a linear subspace with implicitly penalizing curvature.Source: §1 · p. 2
• The proposed algorithm uses weight ensembles to bias optimization toward the monotonicity required by Pareto optimality.The paper also presents a geometric view in which functionally diverse solutions can lie along a straight path in weight space.Source: §1 · p. 2
• Across multiple benchmarks, Pareto Manifold Learning discovers Pareto subspaces and outperforms baselines while producing a more reliable preference-to-objective mapping than other Pareto-front approximations.The conclusion reports performance on a diverse benchmark suite as on par with or better than state-of-the-art MTL methods.Source: §1 · p. 2

2. Related Work

• MTL methods combine architectural modules [Misra et al., 2016] or adjust descent directions for shared parameters.The paper focuses on optimization methods using a Shared-Bottom architecture and distinguishes loss- from gradient-balancing approaches.Source: §2 · p. 2
• Pareto-oriented methods include MGDA, discrete fronts requiring separate runs [Lin et al., 2019], and nearby stationary-point searches [Ma et al., 2020].The paper notes that MGDA does not account for task preferences and can bias solutions toward the task with the smallest gradient magnitude.Source: §2 · p. 2
• Navon et al. (2021) and Lin et al. (2021) employ hypernetworks to continuously approximate the Pareto Front in a single run, introducing additional design choices and limited scalability because they require multiple times the target network's parameters.Ruchte & Grabocka (2021) address scalability by augmenting the feature space with the desired trade-off, sacrificing either functional diversity or optimality.Source: §2 · p. 2
• PaMaL treats weight ensembles as a linear-hypernetwork case, using a convex combination of stored parameters instead of generating weights; this geometric inductive bias allows more reliable and scalable Pareto Front Learning.Source: §2 · p. 3
• Output ensembles can increase inference cost with member count [Lakshminarayanan et al., 2017], while shared or independent subnetworks still do not provide a continuous Pareto-front approximation.The paper discusses parameter sharing and multi-input, multi-output subnetworks [Havasi et al., 2021].Source: §2 · p. 3

3. Problem Formulation

• The ensemble formulation associates each member with network parameters and task weights, then constrains the subspace to their convex hull.For three tasks, Figure 2 illustrates constructing a model by combining member parameters with sampled coefficients.Source: §3 · p. 3, Figure 2 · p. 4
• A point is Pareto optimal when no other point has losses no worse on every task and strictly different on at least one; the set of such points forms the Pareto front.This definition is introduced because competing objectives often make an optimum for all tasks unattainable.Source: §3 · pp. 3–4
• The vector of task losses is scalarized by a weight vector on the task simplex, which encodes task preferences in the overall loss.For example, the stated preference vector [0.8, 0.2] assigns more importance to the first of two tasks.Source: §3 · p. 4
• A Pareto subspace is a parameter-space convex envelope whose network outputs map to a Pareto front in functional space.The formulation defines the front using the vector losses of models with parameters in that subspace.Source: §3 · p. 4

4. Method

• The method seeks identical-architecture networks whose convex-hull combinations form a continuous Pareto front in objective space.Source: §4 · p. 4

4.1. Pareto Manifold Learning

• Using the same weighting for losses and weight interpolation associates models with task losses and guides learning toward a subspace encoding tradeoffs.Source: §4.1 · p. 4
• Each training sample interpolates network weights with α and scalarizes the vector of task losses with the same α.The procedure repeats the forward pass W times per batch and adds a regularizer for non-Pareto-stationary points.Source: §4.1 · p. 4
• The convex hull of the optimal ensemble parameters is a Pareto Subspace.Source: §4.1 · p. 4

14 end

• In practice, the subspace is uniformly sampled, and Pareto optimality is relaxed to require the characterization only in an open neighborhood.Source: §14 · p. 5
• Overparameterization and the universal approximation theorem support the claim that the Pareto front admits a linear parameterization.Source: §14 · p. 5

4.2. Regularization and balancing

• Different task-loss scales can favor easier tasks and undermine scale invariance, motivating loss and gradient balancing schemes that warp loss-weight space.Loss balancing also affects task-specific decoders, while gradient balancing applies to shared parameters.Source: §4.2 · p. 5
• Loss balancing normalizes each task loss using past values, while gradient balancing normalizes each task gradient to unit ℓ2 norm before combining them.Source: §4.2 · p. 5
• Repeating the forward pass W times with different random weightings advances ensemble members concurrently and reduces the variance of pernicious updates.Source: §4.2 · p. 5
• Multi-forward regularization penalizes sampled interpolations when their task losses violate the monotonic ordering imposed by their weights.The penalty is zero when there are no violations.Source: §4.2 · pp. 5–6, Figure 3 · p. 6
• The regularizer keeps one outgoing edge per node in task lexicographic order because the original edge set heavily penalizes connections from low-valued vertices.Source: §4.2 · p. 6
• A symmetric Dirichlet sampling distribution with p ≥ 1 guides the ensemble toward representations suited for multi-task learning; p controls concentration from the simplex center to uniform sampling.Source: §4.2 · p. 6

5. Experiments

• The experiments span MultiMNIST, Census, MultiMNIST-3, UTKFace, and CityScapes, using architectures from MLPs through CNNs to ResNets and including regression and three-task settings.The three-task results are discussed qualitatively in the main text, with quantitative findings in the appendix.Source: §5 · p. 6, §5.2 · p. 8

5.1. Classification on MultiMNIST and Census

• All PaMaL seeds find functionally diverse solutions with higher HyperVolume than baselines, while most baselines show limited diversity and do not consistently outperform Linear Scalarization.The reported comparison uses multi-forward regularization with W = 4, λ = 0 for MultiMNIST and W = 2, λ = 5 for Census.Source: §5.1 · p. 7
• On Census, where tasks differ in statistics, PaMaL recovers diverse Pareto solutions; COSMOS has an invalid preference mapping, while PHN yields functionally limited solutions and uses about 100× more parameters.The authors attribute difficulties with PHN and COSMOS partly to complex mappings from user tradeoffs to model weights.Source: §5.1 · pp. 7–8

5.2. Beyond Pairs of Classification Tasks: MultiMNIST-3 and UTKFace

• For MultiMNIST-3, the three-task subspaces show gradual transfer of learned representation between members, with task symmetry motivating the use of no balancing scheme.The authors describe the 3D plot and simplices as visual confirmation of Theorem 3.Source: §5.2 · p. 8
• On UTKFace, which combines age regression with gender and ethnicity classification, PaMaL discovers a Pareto subspace despite the dataset and network complexity and the different loss scales.The authors use gradient balancing; most of the subspace gives high performance and implicitly low loss for each task, with a dynamic transition in weighted loss as α varies.Source: §5.2 · pp. 8–9

5. Experiments

• On MultiMNIST and Census, PaMaL discovers diverse Pareto-optimal subspaces and outperforms baselines; on CityScapes, it is optimal with MGDA for depth and outperforms most methods for segmentation.For CityScapes, MGDA performs poorly on segmentation, while PaMaL offers a balanced solution; PHN is omitted because it does not scale to large networks.Source: Figure 4 · p. 7, §5.3 · p. 9

6. Conclusion

• PaMaL trains multiple single-task predictors together to form a convex-hull subspace with Pareto properties, and experiments show it outperforms or matches state-of-the-art MTL methods across diverse benchmarks.Hierarchical weight ensembling and connections to task affinity are identified as future directions.Source: §6 · p. 9

A.1. Effect of sampling on the Pareto properties of the discovered subspace

• For two tasks, p < 1 concentrates Dirichlet sampling toward ensemble members, whereas p > 1 concentrates it toward the equal-weight midpoint; extreme values can undermine shared or diverse representations.The text describes p → 0 as yielding independent single-task predictors and very large p as causing members to collapse.Source: §A.1 · p. 14
• On MultiMNIST and Census, p = 0 yields separate single-task predictors without shared representation, while p = 100 can collapse members; p = 0.1 retains diversity but can make interpolated middle models suboptimal.For Census at p = 100, the collapse is accompanied by cosine similarity cos(θ1, θ2) > 0.9.Source: §A.1 · p. 15

A.2. Connection between Pareto Optimality and multiple valley intersections

• As α varies, PaMaL tracks an interpolated model along a changing weighted-loss objective; the authors distinguish this dynamic weighting scenario from simply seeking an intersection of low-loss landscapes.The illustrative objective is L(θ, α) = αL1(θ) + (1 − α)L2(θ).Source: §A.2 · pp. 15–16

A.3. Proof of Theorem 4

• For any compact A and finite family of continuous mappings, a ReLU multilayer perceptron has two parameterizations whose interpolation approximates each mapping uniformly within any chosen ε.For each mapping fn, some α in [0, 1] gives |fn(x) − f(x; αθ + (1 − α)θ′)| ≤ ε for all x in A.Source: §A.3 · p. 17
• The construction places differing task-specific values in U1 and U0 while keeping the other listed parameters shared, so interpolating the parameterizations interpolates those values.The displayed identity gives f(x; αθ + (1 − α)θ′) = f(x; R, αU1 + (1 − α)U0, MS, B, M′).Source: §A.3 · p. 17
• The appendix illustrates the Multi-Forward Graph for two tasks with a window W = 5 and describes a three-task graph whose edges follow each task’s total ordering.Dashed edges are removed from the final graph.Source: Figure 9 · p. 18, Figure 10 · p. 18

B.1. Ablation on Multi-Forward Regularization

• Multi-forward regularization penalizes interpolated models when their sampled losses violate the task ordering implied by their interpolation weights.W counts the α resamplings per batch; λ controls regularization strength.Source: §B.1 · p. 18
• For MultiMNIST, λ = 0 is beneficial because the symmetric tasks share helpful features and produce few monotonicity violations; severe regularization can hinder training.With λ = 0, the subspace is still sampled W times and losses from the interpolated models are included.Source: §B.1 · p. 18
• W = 4 and λ = 0 give the best mean HyperVolume on MultiMNIST, while W = 2 and λ = 5 do so on Census.The accompanying figures report that multi-forward regularization can help on Census, but widening the window may hinder performance unless regularization is stronger.Source: §B.1 · p. 18, Table 2 · p. 19, Table 3 · p. 19

B.2. Illustrative example: ablation on loss/gradient balancing schemes

• With unequal loss scales, loss balancing recovers the exact Pareto Front for all but five initialization pairs, while gradient balancing recovers a superset across all pairs.Without balancing, the method tends toward a subset influenced by the higher-magnitude task; equal-scale runs recover the exact front in most pairs.Source: §B.2 · p. 23, Figure 15 · p. 25, Figure 16 · p. 26

B.3. UTKFace: ablation on the effect of loss/gradient balancing schemes

• On UTKFace, gradient balancing improves functional diversity and overall performance, including a significant improvement in age-prediction Huber loss.Table 4 compares PaMaL without balancing, with gradient balancing, and with loss balancing.Source: §B.3 · p. 28, Table 4 · p. 28

B.4. Hyperparameter optimization for PHN and COSMOS

• PHN hyperparameters are selected by average HyperVolume subject to a Spearman-correlation threshold, because unconstrained selection yielded degenerate solutions.The threshold is intended to favor tradeoff-conditioned solutions whose task performances move oppositely; Figure 19 shows the degenerate cases without it.Source: §B.4 · p. 28, Figure 19 · p. 32
• The PHN sweep varies Dirichlet concentration and solver, while the COSMOS sweep varies concentration, regularization strength, and learning rate.The reported sweeps cover MultiMNIST and Census; PHN’s large hypernetwork requires chunking for UTKFace and CityScapes, and CityScapes PHN results are omitted after chunking failed to yield good results.Source: §B.4 · pp. 28, 30

C.1. Details on experimental configurations

• MultiMNIST is built from two sampled MNIST digits placed on a 36 × 36 grid and resized to 28 × 28, using a shared-bottom LeNet trained for 10 epochs.Census uses age and education-level tasks with a shared-bottom MLP; UTKFace uses age, gender, and ethnicity tasks with a ResNet18 encoder, while CityScapes uses segmentation and depth tasks with SegNet.Source: §C.1 · p. 34

C.2. HyperVolume analysis on MultiMNIST and Census

• HyperVolume measures exploration quality; Table 9 reports that the best results are concentrated in PaMaL rows, with three-decimal reporting producing ties.The table presents classification accuracy for both tasks and HyperVolume, with three seeds per method; Figure 20 illustrates the metric using rectangles from a reference point to sampled points.Source: Table 9 · p. 35, §C.2 · p. 35, Figure 20 · p. 35

C.3. MultiMNIST-3 quantitative results

• Each MultiMNIST-3 sample combines three randomly sampled MNIST digits on a 42 × 42 grid, resolves overlaps by taking the maximum pixel value, then resizes to 28 × 28.The digits occupy the top-left, top-right, and bottom-middle positions.Source: §C.3 · p. 36
• Table 10 compares mean accuracy and standard deviation over three seeds, reporting PaMaL’s best interpolated-model performance without balancing or regularization.Bold marks the best-performing multi-task method; the supplied table passage does not state the cell values.Source: Table 10 · p. 36, §C.3 · p. 36

• The dataset has no publicly available test set, and prior MTL work does not clarify training-set splits for hyperparameter selection.Source: §C.4 · p. 38
• Setting I uses all 2,975 training images without validation; Setting II reserves 500 for validation and hyperparameter tuning, and both use the same test set.Source: §C.4 · p. 38
• MGDA performs optimally among MTL methods for depth estimation but poorly for segmentation, COSMOS favors segmentation, and PaMaL produces balanced solutions.Source: §C.4 · p. 38