PaperlayerView exact revision on arXiv

Source-linked AI summary

Scalable Pareto Front Approximation for Deep Multi-Objective Learning

Michael Ruchte, Josif Grabocka

arXiv:2103.13392v2cs.LG

TL;DR

Deep-learning MOO lacks a scalable way to generate Pareto fronts without training a separate network for each solution or using parameter-heavy hyper-networks. COSMOS conditions a network on preferences in feature space and adds a penalty to encourage a well-spread front. Experiments report strong quality relative to state-of-the-art methods, significantly faster computation, and 7% training-time overhead versus single-objective optimization.

  • Problem

    Existing deep-learning MOO approaches train one network per Pareto-front solution or use parameter-heavy hyper-networks, and scalable generation of a front for practitioner-selected trade-offs remains unavailable.

    Source: §2 · p. 2, §8 · p. 8

  • Method

    COSMOS augments the feature space with preference vectors to condition one prediction network and penalizes solutions to maintain a small angle to the preferences.

    Source: §1 · p. 1

  • Results

    7% training-time overhead versus single-objective optimization is reported, while the method competes strongly with state-of-the-art MOO quality and is significantly faster.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported experiments show that a deep model can approximate a Pareto front on large multi-objective datasets with a small training-time overhead.

    Source: §1 · p. 2

    COSMOS produces an approximation rather than the exact optimum of linear scalarization, and preference-conditioned prediction cannot model objectives independent of the input, such as an Lp weight regularizer.

    Source: §7 · p. 8

Abstract

from arXiv · show

Multi-objective optimization (MOO) is a prevalent challenge for Deep Learning, however, there exists no scalable MOO solution for truly deep neural networks. Prior work either demand optimizing a new network for every point on the Pareto front, or induce a large overhead to the number of trainable parameters by using hyper-networks conditioned on modifiable preferences. In this paper, we propose to condition the network directly on these preferences by augmenting them to the feature space. Furthermore, we ensure a well-spread Pareto front by penalizing the solutions to maintain a small angle to the preference vector. In a series of experiments, we demonstrate that our Pareto fronts achieve state-of-the-art quality despite being computed significantly faster. Furthermore, we showcase the scalability as our method approximates the full Pareto front on the CelebA dataset with an EfficientNet network at a tiny training time overhead of 7% compared to a simple single-objective optimization. We make our code publicly available at https://github.com/ruchtem/cosmos.

1. Introduction

The paper addresses scaling deep-learning MOO beyond approaches that train a separate network per Pareto-front solution or rely on parameter-heavy hyper-networks. COSMOS conditions a network on preference vectors in feature space and adds a penalty to spread solutions; experiments report competitive quality and 7% training-time overhead versus single-objective optimization.

1. Introduction

  • MOO seeks non-dominating solutions that expose trade-offs among objectives, from which practitioners select a post hoc solution.A Pareto-front solution cannot be further optimized without sacrificing performance on at least one objective.Source: §1 · p. 1
  • Prior deep-learning approaches either train a new network for each Pareto-front solution or use hyper-networks that add substantial trainable-parameter overhead.Both approaches are presented as barriers to scaling to deep neural networks.Source: §1 · p. 1
  • COSMOS conditions a prediction model on preference vectors by augmenting the feature space, allowing one network to approximate the full Pareto front in a single optimization run.This avoids the need to train one network per front solution and does not significantly increase trainable parameters as hyper-networks do.Source: §1 · pp. 1–2
  • A penalty term encourages solutions produced through linear scalarization to remain at a small angle to preference vectors, promoting a well-spread Pareto front.Linear scalarization is described as fast for gradient-based learning but not well-spread without this remedy.Source: §1 · pp. 1–2
  • 7% training-time overhead versus single-objective optimization is reported for approximating Pareto fronts for deep models on large multi-objective datasets.The paper also reports that the method competes strongly with state-of-the-art MOO methods in quality while being significantly faster.Source: §1 · p. 2

2. Related Work

Related work covers multi-task learning and other multi-objective applications, then contrasts scalable deep-learning MOO with approaches that train networks per solution or rely on parameter-heavy hyper-networks. COSMOS instead conditions feature representations on preferences and adds a penalty to improve front quality.

Multi-Task Learning and Fairness

  • Multi-task learning jointly trains a prediction model for multiple tasks, while fairness MOO jointly balances objectives based on the same output.The related work notes that these areas use distinct mechanisms for task prioritization, loss balancing, architecture, or uncertainty.Source: §2 · p. 2

Multi-Objective Optimization

  • Earlier gradient-based approaches find a single Pareto-stationary solution, while follow-up methods align multiple solutions to preference rays but train a separate network for each point.Weight transfer between solutions is described as one strategy for speeding their generation.Source: §2 · p. 2

Conditional Multi-Objective Optimization

  • Hyper-network approaches condition model weights on preferences, while COSMOS conditions the feature space and adds a penalty intended to improve Pareto-front quality under linear scalarization.The related work describes hyper-networks as predicting another network's weights from an input.Source: §2 · p. 2

Deep Multi-Objective Optimization

  • Prior deep-learning MOO methods either train multiple networks to populate a Pareto front or depend on parameter-heavy hyper-networks, whereas COSMOS aims for comparable single-network optimization time.The paper also situates MOO applications in reinforcement learning and neural architecture search.Source: §2 · p. 2

3. Preliminary

This section defines Pareto-optimal solutions and the Pareto front for jointly minimizing multiple objectives. It also explains how linear scalarization approximates Pareto-optimal solutions and why repeating it across preferences is costly.

3. Preliminary

  • A solution is Pareto optimal when no other solution dominates it, and the Pareto front contains the objective values of all such solutions.Dominance means being no worse on every objective and better on at least one.Source: §3 · p. 3
  • Linear scalarization treats multi-objective optimization as a single-objective problem using a preference vector, but first-order optimization yields only approximate Pareto-optimal solutions.For neural networks, the loss functions are non-convex with respect to their weights.Source: §3 · p. 3
  • Approximating the front by solving for many preference vectors requires repeating the scalarized optimization, which is computationally intractable when the number of desired points is large.Source: §3 · p. 3

4. Proposed Method

The method conditions one neural network on preference vectors to approximate a Pareto front in a single optimization run. It combines preference-weighted loss minimization with a cosine-similarity penalty and uses one back-propagation for the summed losses.

4. Proposed Method

  • The model receives the preference vector alongside the input, with preferences sampled from a Dirichlet distribution over the simplex.The conditioned model predicts solutions corresponding to the input preference vectors.Source: §4 · p. 3
  • A single conditioned neural network approximates the Pareto front in one gradient-based optimization run, instead of training a new network for each preference.Source: §4 · p. 3
  • To address narrow fronts from linear scalarization, the objective minimizes preference-weighted losses while maximizing cosine similarity between preferences and losses.The penalty weight λ controls the regularization magnitude.Source: §4 · p. 4
  • The optimization has complexity independent of the number of objectives and needs only one back-propagation through the network for the sum of losses.The paper describes its runtime as asymptotically comparable to a single-objective problem.Source: §4 · p. 4

5. Experimental Protocol

The experiments assess COSMOS against multi-objective and single-objective baselines, test its cosine-similarity penalty, and examine scaling to CelebA. The protocol separates small-scale quality comparisons from a deep-network scalability test.

5.1. Research Hypotheses and Designed Experiments

  • Experiments compare COSMOS with gradient-based Pareto-front baselines on fairness and image-classification datasets, and with single-objective baselines on the same datasets.The Pareto-front baselines are PHN-LS, PHN-EPO, and ParetoMTL; the datasets include Adult, Compass, Default, and three Multi-MNIST variants.Source: §5.1 · p. 4
  • An ablation varies λ and α on image-classification datasets to examine their effect on Pareto-front spread.This tests whether the cosine-similarity penalty produces a wider-spread front.Source: §5.1 · p. 4

5.2. Experimental Setup

  • The setup separates small-scale baseline-quality comparisons from a CelebA scalability test because prior methods cannot scale to the deep-network experiment.The published baseline experiments used toy-scaled LeNet; on CelebA, ParetoMTL requires a separate EfficientNet per front point, while PHN-LS and PHN-EPO use approximately 100× EfficientNet's trainable parameters.Source: §5.2 · p. 5

5.3. Reproducibility

  • Baseline hyperparameters follow their papers and official implementations, with validation hyper-volume early stopping and mean scores and fronts reported over five runs except on CelebA.The general protocol uses batch size 256, Adam at 10^-3, reference point (2, 2), and 25 equally distributed test rays.Source: §5.3 · p. 5
  • Fairness experiments use sex as the binary sensitive attribute and a hyperbolic-tangent relaxation of Difference of Equality of Opportunity as a differentiable objective.The tabular setup uses an MLP with hidden dimensions 60 and 25, 70/10/20 train-validation-test splits, and dataset-specific α values for Default.Source: §5.3 · pp. 5–6
  • Multi-MNIST experiments use LeNet with task-specific heads, cross-entropy losses for the two digit tasks, and preference-vector features concatenated with the image.The preference vector is transformed with transposed convolutions into feature maps that are upsampled and appended as image channels.Source: §5.3 · p. 6
  • The CelebA experiment uses an ImageNet-pretrained EfficientNet-B4 with four binary-cross-entropy objectives spanning two easy and two hard attributes.Images are rescaled to 64 × 64; the objectives are Goatee, Mustache, Oval Face, and Pointy Nose.Source: §5.3 · p. 6

6. Results

COSMOS produces competitive Pareto fronts across the reported datasets and approximates a full front on CelebA with a small training-time overhead. The experiments also examine convergence and how the penalty and sampling parameter affect the fronts.

Fairness on tabular data and Multi-MNIST

  • Figure 3 tracks COSMOS Pareto-front convergence on image-classification datasets; the fronts are reported as well-spread after 10 epochs.Source: Figure 3 · p. 7, §6 · p. 7

Ablation study

  • Increasing the cosine penalty widens the front, while α affects front quality rather than width; α < 1 samples more near simplex vertices.The ablation varies α over 0.1, 0.7, and 1.2, and λ over 1, 3, and 5.Source: Figure 4 · p. 7, §6 · p. 7

Fairness on tabular data and Multi-MNIST

  • COSMOS is not outperformed by PHN-EPO, PHN-LS, or ParetoMTL on any of six datasets in hyper-volume.The reported results describe COSMOS as comparing favorably to baselines on Pareto-front quality.Source: §6 · pp. 6–7
  • On all three Multi-MNIST datasets, COSMOS outperforms the single-task baseline.The comparison is for this multi-task problem.Source: §6 · p. 7

CelebA

  • 7% additional training time yields a full Pareto front instead of a single point, with both methods run for 25 epochs.The comparison is against the single-task baseline.Source: §6 · p. 8
  • The pretrained ImageNet EfficientNet produces a well-spread CelebA front within a few epochs and converges entirely after 25 epochs.Figure 5 presents fronts for the easy and hard tasks.Source: §6 · p. 7, Figure 5 · p. 8

7. Limitations of our method

COSMOS provides approximate rather than exact Pareto fronts, and preference conditioning cannot model objectives that do not depend on the input.

7. Limitations of our method

  • COSMOS fronts are approximations, and preference conditioning cannot model input-independent objectives such as an Lp weight regularizer.Source: §7 · p. 8

8. Conclusions

COSMOS conditions a prediction model on objective preferences and uses a penalty to spread the resulting Pareto front. The authors report competitive fronts with low training-time overhead, including a CelebA EfficientNet result with 7% overhead.

8. Conclusions

  • COSMOS augments the feature space with preference vectors so one network adjusts predictions to the selected trade-off.Source: §8 · p. 8
  • A penalty that minimizes the angle between achieved objective-space solutions and preference vectors encourages widely spread fronts.Source: §8 · p. 8
  • One optimization run approximates the full Pareto front, with asymptotic optimization cost equal to a single-objective task.Source: §8 · p. 8
  • The approximated fronts are competitive against state-of-the-art baselines, and CelebA EfficientNet requires only 7% training-time overhead over single-objective optimization.Source: §8 · p. 8

A. More objectives on CelebA

CelebA experiments extend COSMOS to multiple and randomly selected objectives, while additional illustrations explain the cosine-similarity penalty and report fairness comparisons and ablations.

A. More objectives on CelebA

  • The four-task CelebA setting achieves performance similar to the single-task baseline, while its MCRs describe only one Pareto-front point and hypervolume is not reported for more than three objectives.The tasks are Goatee, Mustache, No Beard, and Pale Skin; evenly distributed points on a (J −1)-sphere were considered nontrivial to obtain for J > 3.Source: A. More objectives on CelebA · p. 11, Table 3 · p. 11
  • The three-objective COSMOS front uses 25 test rays evenly distributed on a 2-sphere via the Fibonacci sphere algorithm.Figure 6 shows the front for the first three objectives.Source: A. More objectives on CelebA · p. 11, Figure 6 · p. 11
  • COSMOS was also evaluated on 10 randomly selected CelebA tasks using MCRs against the single-task baseline, with training increased to 30 epochs.The tasks include Black Hair, Wearing Lipstick, Bald, Goatee, Big Nose, Smiling, Receding Hairline, Sideburns, No Beard, and Chubby.Source: A. More objectives on CelebA · p. 11, Table 4 · p. 12

B. Intuition behind the Cosine Similarity penalty

  • A small angle between the loss and preference vectors steers solutions toward the desired position while still minimizing both losses, balancing loss minimization and Pareto-front positioning.The illustrations contrast regular linear scalarization with the desired front and show the effect of angle minimization.Source: B. Intuition behind the Cosine… · p. 11, Figure 7 · p. 12, Figure 8 · p. 12

C. Additional figures

  • Figure 9 compares COSMOS with the single-task baseline on the fairness datasets.The supplied passage identifies the comparison but does not state its outcome.Source: C. Additional figures · p. 12, Figure 9 · p. 13
  • On fairness datasets, the λ and α ablations show behavior like Multi-MNIST, although cosine similarity appears less important for fairness than for multi-task learning.The authors suggest the difference may relate to the large difference in scales.Source: C. Additional figures · p. 13

D. More detailed result tables

The section points to detailed results for fairness and image-classification datasets.

D. More detailed result tables

  • Table 5 presents detailed results on fairness and image-classification datasets.The passage names the table's scope but provides no individual results.Source: D. More detailed result tables · p. 13

E. Experimental Setup (further details)

Further experimental details cover hardware and seeds, dataset dimensions, selected training settings, and CelebA image preprocessing.

E. Experimental Setup (further details)

Related papers

Loading 2103.13392v2…