PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Pareto Manifold Learning with Low-Rank Structure

Weiyu Chen, James T. Kwok

arXiv:2407.20734v1cs.LG

TL;DR

Discrete Pareto-front methods offer only finite trade-off solutions, while approaches using base networks face scalability challenges when task counts grow. This paper combines a main network with low-rank matrices to learn trade-off solutions, and reports better performance than PaMaL and other baselines, especially with many tasks.

  • Problem

    Discrete Pareto-front methods provide only a finite set of trade-off solutions, and maintaining a separate base network per task creates parameter and sharing challenges as task counts grow.

    Source: §1 · p. 1

  • Method

    The method combines a main network with multiple low-rank matrices and uses orthogonal regularization to promote disentanglement.

    Source: §1 · p. 1

  • Results

    The method outperforms PaMaL and other baselines, particularly when the number of tasks is large.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The main network captures shared features while low-rank matrices capture task-specific differences, and the approach approximates the Pareto front with fewer parameters for many tasks.

    Source: §1 · p. 1

    The method uses the same rank for all layers; varying layer ranks and automatic rank selection are left for future work.

    Source: §5 · p. 9

Abstract

from arXiv · show

Multi-task learning, which optimizes performance across multiple tasks, is inherently a multi-objective optimization problem. Various algorithms are developed to provide discrete trade-off solutions on the Pareto front. Recently, continuous Pareto front approximations using a linear combination of base networks have emerged as a compelling strategy. However, it suffers from scalability issues when the number of tasks is large. To address this issue, we propose a novel approach that integrates a main network with several low-rank matrices to efficiently learn the Pareto manifold. It significantly reduces the number of parameters and facilitates the extraction of shared features. We also introduce orthogonal regularization to further bolster performance. Extensive experimental results demonstrate that the proposed approach outperforms state-of-the-art baselines, especially on datasets with a large number of tasks.

1. Introduction

Pareto-front methods offer trade-offs among tasks, but discrete solutions limit preference flexibility and continuous approaches face scalability challenges. The paper proposes a main network with low-rank matrices to reduce parameters and capture shared and task-specific features, reporting stronger performance especially with many tasks.

1. Introduction

  • Multi-task learning balances tasks that may have disparate scales and conflicting objectives while leveraging shared information to improve overall performance.The paper frames this balancing challenge as a multi-objective optimization problem.Source: §1 · p. 1
  • Existing Pareto-front algorithms provide only a finite set of discrete trade-off solutions, limiting adaptation to varying user preferences.Continuous approximation instead aims to provide preference-conditioned solutions at arbitrary resolution, but hypernetwork size can restrict use to small base networks, while COSMOS has limited parameter space.Source: §1 · p. 1
  • PaMaL combines task-specific base networks according to preferences, but maintaining one network per task increases parameters and prevents the networks from learning from one another.These costs pose challenges as the ensemble grows.Source: §1 · p. 1
  • The proposed main-network and low-rank-matrix approach reduces parameter growth for many tasks and outperforms PaMaL and other baselines, especially at large task counts.The main network captures shared features, while low-rank matrices capture task-specific differences; orthogonal regularization promotes their disentanglement.Source: §1 · p. 1

2. Background

The background frames multi-task learning as multi-objective optimization and reviews methods for approximating the Pareto front. It contrasts discrete and continuous approaches and motivates jointly learning low-rank matrices as a model subspace.

2.1. Multi-Objective Optimization

  • A multi-objective problem minimizes a vector of objective functions; a solution is Pareto-optimal when no other solution dominates it across objectives.The Pareto set contains all Pareto-optimal solutions, and the Pareto front is the set of their functional values.Source: §2.1 · p. 2

2.2. Multi-Task Learning

  • MTL uses a shared-bottom network with task-specific heads, and many conflict-handling algorithms seek a single solution rather than the whole Pareto front.Discrete approximation methods generate fixed sets of Pareto-front points; expanding points locally can yield segments rather than the whole continuous front.Source: §2.2 · p. 2
  • Continuous Pareto-front methods condition solutions on user preferences, but hypernetworks can be larger than their base networks and preference-input methods may have limited parameter space.PaMaL combines multiple base networks, but its per-task networks increase parameter count and cannot learn from one another; continuous-optimization approaches cited here fail to converge in deep-learning scenarios with much larger parameter spaces.Source: §2.2 · p. 2

2.3. Low-Rank Adaptation

  • Earlier LoRA work merges modules or fine-tunes a model with one module, whereas the proposed approach jointly trains multiple low-rank matrices to produce a network subspace.The background states that prior work had not considered joint training of an arbitrary convex combination of low-rank matrices to learn a model subspace.Source: §2.3 · pp. 2–3

3. Proposed Method

LORPMAN combines a main network with task-specific low-rank matrices to learn preference-conditioned Pareto solutions, with orthogonal regularization to encourage separation among the matrices.

3.1. Low-Rank Structure

  • PaMaL base-network similarities rise during training and are higher in lower layers, motivating a shared main module plus low-rank task-specific adaptations.The main modules capture common features across tasks, while the low-rank adaptations represent task differences.Source: §3.1 · p. 3

3.1.1. COMPUTATIONAL EFFICIENCY

  • The parameter count is d_lk_l + m(d_lr_l + r_lk_l), versus m d_lk_l for PaMaL, with the efficiency gain growing for larger networks and more tasks.The rank r_l is usually much smaller than k_l and d_l.Source: §3.1.1 · p. 3

3.1.2. APPROXIMATION POWER

  • Under compactness of X × Δm and continuity of t(x, α), a ReLU MLP with a main network and m low-rank matrices approximates the Pareto mapping within any ε > 0.This result applies across inputs x and preference vectors α.Source: §3.1.2 · pp. 3–4

3.2. Orthogonal Regularization

  • Orthogonal regularization encourages the low-rank matrices to be mutually orthogonal, aiming to reduce redundancy and conflicts while separating shared and task-specific features.The matrices are flattened, normalized, and concatenated by layer; when m exceeds 3, a random subset of three tasks is used, making loss complexity independent of m while maintaining satisfactory performance.Source: §3.2 · p. 4

3.3. Optimization

  • Training first adapts the main network and low-rank matrices, then freezes the main network so the matrices learn task-specific representations; preference vectors are sampled from a Dirichlet distribution.The method combines multi-forward and orthogonal losses, and uses linear scalarization, whose optimal solutions are Pareto-optimal.Source: §3.3 · pp. 4–5

4. Experiments

Experiments evaluate LORPMAN on a toy problem and datasets with up to 40 tasks, comparing Pareto-front quality, parameter use, and ablation settings. Across these tests, the reported results favor LORPMAN over continuous and discrete baselines, particularly as task counts grow.

4.1. Illustrative Example

  • On the toy problem, preference vectors [0.5, 0.5], [1, 0], and [0, 1] reach the PF’s middle and two extremes, respectively.The resulting solutions lie on the same line in parameter space, and varying the preference vector can obtain the whole Pareto set.Source: §4.1 · p. 5, Figure 3 · p. 6

4.2. Datasets with Two or Three Tasks

  • On MultiMNIST and Census, LORPMAN obtains a Pareto front closer to the top-right region, while its UTKFace solutions outperform PaMaL in all three objectives.On MultiMNIST and Census, PHN, PHN-HVI, and PaMaL have inferior HVs despite having more parameters than LORPMAN.Source: §4.2 · p. 6, Figure 4 · p. 7, Figure 5 · p. 7

4.3. Scale to Large Number of Tasks

  • On CIFAR-100, LORPMAN achieves significantly better HV using 8.9% of PaMaL’s parameters; on CelebA, it performs better using 21% of PaMaL’s parameters.On CIFAR-100, PaMaL’s slow convergence from jointly training 20 base networks is not resolved by simply increasing its learning rate.Source: §4.3 · pp. 7–8, Table 3 · p. 7, Table 4 · p. 8

4.4. Comparison with Discrete PF Baselines

  • On MultiMNIST, LORPMAN performs better than the compared discrete Pareto-front algorithms, which generate only a predetermined number of solutions.The comparison includes EPO, PMTL, MOO-SVGD, PNG, and GMOOAR.Source: §4.4 · p. 8, Figure 7 · p. 8

4.5. Comparison with Single-Solution Baselines

  • On CityScapes, LORPMAN surpasses COSMOS, PaMaL, and single-solution algorithms overall, although MGDA has slightly better relative depth error and worse segmentation performance.The comparison selects one solution from each continuous Pareto-front method; the paper’s objective is to learn the entire front.Source: §4.5 · pp. 8–9, Table 5 · p. 9

4.6. Ablation Studies

  • Orthogonal regularization reduces correlation between low-rank matrices, while freezing the main model during the latter half of training encourages task-specific representations and better performance.The authors report that the correlation reduction encourages common features in the main network and differences in the low-rank matrices.Source: §4.6 · p. 9, Table 6 · p. 9, Table 8 · p. 9
  • On UTKFace, rank 8 already outperforms COSMOS and PaMaL; rank 64 achieves better performance, while further rank increases yield no significant change, and scaling factors in [0.1, 1] give stable performance.Source: §4.6 · p. 9, Table 7 · p. 9, Table 9 · p. 9

5. Conclusion

The paper introduces a continuous Pareto-front method built from a main network and low-rank matrices, reporting superior performance especially on datasets with many tasks. It identifies using the same rank for every layer as a limitation.

5. Conclusion

  • A main network with low-rank matrices forms the proposed continuous Pareto-front method, which performs especially well when the number of tasks is large.The method is evaluated across various datasets.Source: §5 · p. 9
  • The method uses the Pareto manifold's structure to learn trade-off solutions among tasks.Source: §5 · p. 9
  • The same rank is used for all layers, and different layer-wise ranks and automatic rank setting are future directions.Source: §5 · p. 9

Impact Statement

The paper presents improved performance and cost efficiency in real-world multi-objective applications as potential consequences of discovering trade-off solutions. It also flags potential bias and privacy concerns when processing data across tasks.

Impact Statement

  • The proposed algorithm can discover trade-off solutions among multiple tasks, with potential for improved performance and cost efficiency in real-world multi-objective applications.Source: Impact Statement · p. 10
  • Processing data from different tasks carries potential bias and privacy concerns.Source: Impact Statement · p. 10

A. Proof of Theorem 3.1

The theorem establishes an approximation guarantee for a continuous mapping from network inputs and preference vectors to Pareto-front points. Its proof constructs a ReLU MLP with a main network and low-rank matrices.

A. Proof of Theorem 3.1

  • Under compactness of X × ∆m and continuity of t(x, α), a ReLU MLP with a main network and m low-rank matrices approximates the mapping within ϵ.The mapping takes an input and preference vector to the corresponding Pareto-front point.Source: A. Proof of Theorem 3.1 · p. 13
  • The proof begins with a ReLU MLP that represents g(x, α) using the concatenated input and preference vector.Source: A. Proof of Theorem 3.1 · p. 13
  • The construction defines matrices R and S and vectors U_i to encode the input and preference-dependent components in the network.Source: A. Proof of Theorem 3.1 · p. 13
  • The constructed network is written as h(x; M, N, C, R, S, U), with θ0 collecting the main-network parameters and θi the task-specific Ui component.Source: A. Proof of Theorem 3.1 · p. 13
  • Each θi has one nonzero entry and can be reshaped as a rank-1 matrix BiAi.Source: A. Proof of Theorem 3.1 · p. 13

B. Proof of Proposition 3.2

The proof states that minimizing a positively weighted sum of task objectives yields a Pareto-optimal solution. The following subsection defines multi-forward regularization using directed edges between preference vectors ordered by a task's preference coordinate.

B. Proof of Proposition 3.2

  • For positive preference weights, an optimizer of the scalarized sum of objectives is Pareto-optimal; the proof rules out domination by contradiction.Source: B. Proof of Proposition 3.2 · p. 14

C. Multi-Forward Regularization

  • Multi-forward regularization constructs a directed graph for each task from sampled reference vectors, with edges ordered by that task's preference coordinate.The passage introduces the regularization loss but does not specify its formula in the supplied text.Source: C. Multi-Forward Regularization · p. 14

D.1. Hypervolume

The section defines hypervolume as a measure of a solution set’s performance and specifies the reference points used for evaluation.

D.1. Hypervolume

  • Hypervolume measures the performance of an obtained solution set relative to a reference point.It is a popular indicator in multi-objective optimization.Source: §D.1 · p. 14
  • HV is the Lebesgue measure of the region bounded by the solution set and reference point.In the two-objective maximization illustration, this region is the shaded area.Source: §D.1 · p. 14
  • Evaluation uses an all-zero reference point except for UTKFace, which uses [0.5, 0, 0] because its first objective is Huber loss.For the other datasets, zero is the smallest possible accuracy; 0.5 is close to the largest observed Huber loss in the UTKFace experiment.Source: §D.1 · p. 14

D.2. Toy Problem

The appendix defines a two-objective toy problem and documents network architectures and parameter settings for experiments across six datasets.

D.2. Toy Problem

  • The two-objective toy problem uses θ = [θ1, θ2] ∈ R2, with each objective coefficient defined by a clipped tanh of θ2.Figure 8 illustrates the hypervolume indicator.Source: §D.2 · p. 14, Figure 8 · p. 15

D.3. MultiMNIST

  • MultiMNIST uses a LeNet shared bottom with two task heads; PHN hypernetworks map preference vectors through an MLP to generate base-network parameters.LORPMAN and PaMaL use λp = 0 and b = 4; the learning rate is 1e−3 and batch size is 256.Source: §D.3 · p. 15

D.4. Census

  • Census uses a one-layer MLP shared bottom and one fully connected layer per task head, with the MultiMNIST hypernetwork structure.LORPMAN and PaMaL use λp = 5 and b = 2; the learning rate is 1e−3 and batch size is 256.Source: §D.4 · p. 15

D.5. UTKFace

  • UTKFace uses a ResNet-18 shared bottom and three fully connected task heads.LORPMAN and PaMaL use λp = 1 and b = 3; the learning rate is 1e−3 and batch size is 256.Source: §D.5 · p. 15

D.6. CIFAR-100

  • CIFAR-100 comprises 20 five-way classification tasks and uses a VGG-16 shared bottom with 20 fully connected task heads.LORPMAN and PaMaL use λp = 1 and b = 2; experiments use SGD with learning rate 1e−2 and batch size 64.Source: §D.6 · p. 16

D.7. CelebA

  • CelebA has 40 binary classification tasks and uses a ResNet-18 shared bottom with 40 fully connected task heads and dropout regularization.LORPMAN and PaMaL use λp = 1 and b = 3; the learning rate is 1e−3 and batch size is 128.Source: §D.7 · p. 16

D.8. Cityscapes

  • Cityscapes uses a SegNet shared bottom with two task heads, each comprising two convolution layers.LORPMAN and PaMaL use λp = 1 and b = 3; the learning rate starts at 1e−4 and is halved after 75 epochs, with batch size 8.Source: §D.8 · pp. 16–17

E. Pareto Front Obtained on CIFAR-100

On CIFAR-100, LORPMAN achieves good accuracy and diversity, while PaMaL has lower accuracies and COSMOS concentrates its solutions in a small region.

E. Pareto Front Obtained on CIFAR-100

Related papers

Loading 2407.20734v1…