PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Task Learning as Multi-Objective Optimization

Ozan Sener, Vladlen Koltun

arXiv:1810.04650v2cs.LGstat.ML

TL;DR

MTL commonly combines task losses linearly even though competing objectives make that compromise inadequate. This paper formulates MTL as multi-objective optimization, develops an efficient upper-bound method, and reports stronger performance than baselines across tasks ranging from 2 to 40. Its method’s computational applicability is bounded by the assumptions and gradient-computation issues identified in the supplied passages.

  • Problem

    Weighted sums of task risks are valid only when tasks do not compete, whereas conflicting MTL objectives require modeling their trade-off.

  • Method

    The paper uses a Frank-Wolfe-based optimizer for multi-objective MTL and optimizes an upper bound with a single backward pass without explicit task-specific gradients.

  • Results

    The method clearly outperforms all baselines on MultiMNIST, CelebA, and Cityscapes experiments with 2 to 40 tasks.

  • Takeaways & Limitations

    The experiments indicate that efficient multi-objective optimization is effective across a wide range of multi-task scenarios.

  • Takeaways & Limitations

    The direct gradient-based MTL update requires one backward pass per task, making training time scale linearly with the number of tasks and potentially prohibitive beyond a few tasks.

Abstract

from arXiv · show

In multi-task learning, multiple tasks are solved jointly, sharing inductive bias between them. Multi-task learning is inherently a multi-objective problem because different tasks may conflict, necessitating a trade-off. A common compromise is to optimize a proxy objective that minimizes a weighted linear combination of per-task losses. However, this workaround is only valid when the tasks do not compete, which is rarely the case. In this paper, we explicitly cast multi-task learning as multi-objective optimization, with the overall objective of finding a Pareto optimal solution. To this end, we use algorithms developed in the gradient-based multi-objective optimization literature. These algorithms are not directly applicable to large-scale learning problems since they scale poorly with the dimensionality of the gradients and the number of tasks. We therefore propose an upper bound for the multi-objective loss and show that it can be optimized efficiently. We further prove that optimizing this upper bound yields a Pareto optimal solution under realistic assumptions. We apply our method to a variety of multi-task deep learning problems including digit classification, scene understanding (joint semantic segmentation, instance segmentation, and depth estimation), and multi-label classification. Our method produces higher-performing models than recent multi-task learning formulations or per-task training.

1 Introduction

Multi-task learning can exploit shared structure, but competing task objectives make weighted loss combinations inadequate. The paper therefore formulates MTL as multi-objective optimization and develops an efficient Pareto-oriented method evaluated across several tasks.

  • Motivation: Shared data and inductive bias can improve performance across jointly learned tasks, even when the tasks appear unrelated.The motivation traces this potential to dependencies arising from shared real-world processes.
  • Problem: Weighted sums of task risks are appropriate only when a parameter set performs effectively across all tasks, a condition that is rarely met.When objectives conflict, MTL must model trade-offs beyond linear combination.
  • Approach: The paper casts MTL as finding solutions that are not dominated by any others, namely Pareto optimal solutions.
  • Computational challenge: MGDA can find Pareto-set points for deep-network MTL but scales poorly with gradient dimensionality and requires one backward pass per task.These costs create technical barriers for large-scale learning.
  • Proposed method: A Frank-Wolfe-based optimizer uses an upper bound computable with a single backward pass and yields Pareto optimal solutions under realistic assumptions.The paper presents this as an exact deep-network multi-objective algorithm with negligible computational overhead.
  • Evaluation: The method clearly outperforms all baselines across MultiMNIST, CelebA, and Cityscapes evaluations spanning 2 to 40 tasks.

2 Related Work

Prior MTL work largely relies on parameter sharing and weighted-sum meta-algorithms, while related multi-objective methods target other model classes or applications. The paper addresses the gap of applying multi-objective optimization to high-capacity gradient-based deep networks.

  • Multi-task learning: MTL commonly uses hard or soft parameter sharing to couple task-specific and shared parameters.
  • Multi-task learning: Existing MTL algorithms commonly use weighted summation as the meta-algorithm for updating shared parameters.Other approaches include task affinity matrices, shared dictionaries, swarm optimization, and heuristic weighting.
  • Research gap: The related methods described do not apply to gradient-based learning of high-capacity models such as modern deep networks.
  • Multi-objective optimization: Gradient-based multi-objective optimization uses multi-objective KKT conditions to find descent directions that decrease all objectives.
  • Multi-objective optimization: Prior multi-objective applications include multi-agent learning, kernel learning, sequential decision making, and Bayesian optimization.

3 Multi-Task Learning as Multi-Objective Optimization

The paper formulates multi-task learning as multi-objective optimization, replacing weighted-loss minimization with Pareto-oriented updates for potentially conflicting tasks. It develops MGDA-UB, an efficient upper-bound method for high-dimensional deep networks that requires a single backward pass and has Pareto-optimality guarantees under stated assumptions.

  • Formulation: MTL uses shared and task-specific parameters, but weighted empirical-risk minimization is valid only when one parameter set performs effectively across all tasks.Conflicting tasks can prefer different solutions, making comparison impossible without task-importance weights.
  • Formulation: The paper instead represents MTL as multi-objective optimization over a vector-valued loss, targeting Pareto optimality rather than a single weighted sum.A Pareto-optimal solution is not dominated by another solution that is no worse on every task and strictly better on at least one.
  • Multiple Gradient Descent Algorithm: MGDA uses KKT conditions to combine task gradients into a shared-parameter update that either satisfies Pareto stationarity or provides a descent direction improving all tasks.The optimization is equivalent to finding a minimum-norm point in the convex hull of task gradients.
  • Solving the Optimization Problem: Frank-Wolfe solves the convex constrained gradient-combination problem efficiently in high-dimensional parameter spaces, where computational-geometry methods designed for low dimensions do not apply.For two tasks, the line search has an analytical solution; the resulting method updates task-specific parameters separately and shared parameters using the combined gradient.
  • Efficient Optimization for Encoder-Decoder Architectures: The original MGDA update is costly because it requires one shared-parameter backward pass per task, causing training time to scale linearly with the number of tasks.MGDA-UB replaces shared-parameter gradients with representation gradients, enabling an upper-bound computation through a single backward pass.
  • Efficient Optimization for Encoder-Decoder Architectures: Under realistic assumptions, optimizing MGDA-UB yields a Pareto-optimal solution or Pareto-stationary point with negligible computational overhead for encoder-decoder deep networks.The guarantee relies on a full-rank representation Jacobian; the paper presents the method as applicable to high-capacity models and many tasks.

4 Experiments

Experiments across digit classification, multi-label classification, and scene understanding test MGDA-based multi-task learning against weighted-sum, single-task, and other baselines. The method handles competing tasks and many-task settings while retaining strong performance and reducing approximation-related training cost.

  • MultiMNIST: MultiMNIST tasks compete for model capacity, so static loss scaling performs below single-task training, whereas the proposed method matches single-task accuracies.Uncertainty weighting and GradNorm remain below the single-task baseline, while the proposed method efficiently uses shared capacity.
  • Multi-Label Classification: On CelebA, the proposed method outperforms baselines for most of the 40 attribute tasks and achieves comparable performance on the remainder.The experiment treats each of 40 attributes as a separate binary classification task and reports per-task errors with a radar chart plus category means.
  • Scene Understanding: Scene understanding jointly evaluates semantic segmentation, instance segmentation, and monocular depth estimation using shared encoder and task-specific decoders.The instance-segmentation output is evaluated through a center-of-mass proxy task, with pairwise performance profiles shown for the jointly trained three-task system.
  • Experiments: Across experiments, multi-task learning outperforms single-task accuracy, and the proposed method outperforms all baselines on every evaluated scene-understanding task.The evaluation spans MultiMNIST, CelebA, and Cityscapes, with task counts ranging from 2 to 40.
  • Role of the Approximation: The MGDA-UB approximation reduces training time by 40% for three scene-understanding tasks and accelerates 40-task multi-label learning by a factor of 25.These runtime measurements compare the proposed algorithm with and without the approximation on a single Titan Xp GPU.
  • Role of the Approximation: The approximation does not cause an accuracy drop and is associated with higher computational efficiency and stability.The reported comparison finds similar or slightly improved accuracy relative to the full method under the stated full-rank assumption.

5 Conclusion

The paper presents an efficient multi-objective approach to multi-task learning and reports effectiveness across diverse task combinations. Performance profiles cover MultiMNIST and Cityscapes, including joint segmentation, instance segmentation, and depth estimation.

  • The approach casts multi-task learning as multi-objective optimization and uses an efficient algorithm with almost no computational overhead.
  • Figure 3 shows that the tasks compete for model capacity and that the proposed method reaches performance as good as dedicated models for each task.
  • MultiMNIST performance is summarized against MTL algorithms and dedicated single-task baselines in Table 3.
  • Cityscapes performance is reported for joint semantic segmentation, instance segmentation, and depth estimation in Table 4.
  • Figure 4 visualizes pairwise projections of Cityscapes performance profiles, while every plotted point solves all three tasks jointly.

A Proof of Theorem 1

The proof establishes equivalence between the original MGDA formulation and its upper-bound optimization under a full-rank assumption. It then uses descent and optimality conditions to establish the theorem’s second case.

  • The proof first shows that a zero optimum for MGDA-UB implies a zero optimum for the original MGDA objective.
  • Under the full-rank assumption on ∂Z/∂θsh, the two formulations agree bidirectionally on Pareto stationarity.
  • The second case requires showing that the MGDA-UB descent direction does not increase any loss function.
  • The proof derives the required result from the optimality conditions of MGDA-UB, using its Lagrangian and KKT condition.

B Additional Results on Multi-label Classification

The additional results report per-attribute binary classification errors for all algorithms in the CelebA multi-label classification experiment. These tabulated results complement the main-text radar chart.

  • The supplement lists binary classification error for each attribute and each algorithm in Table 5.
  • The tabulated results were omitted from the main text because of space limitations, which instead presents a radar chart.
  • Table 5 reports multi-label classification error separately for every attribute.

C.1 MultiMNIST

The MultiMNIST experiments overlay two randomly selected digit images and use a shared LeNet-based encoder with separate task heads. Training uses cross-entropy losses for the left and right digit tasks.

  • MultiMNIST overlays two randomly selected images, placing one at the top-left and the other at the bottom-right.
  • The architecture used for the MultiMNIST experiments is visualized in Figure 5.
  • The MultiMNIST architecture uses a LeNet-based shared encoder followed by independent fully connected layers for the left and right tasks.
  • Each MultiMNIST task uses cross-entropy loss with a softmax output.
  • Models use SGD with momentum, batch size 256, 100 training epochs, and learning-rate halving every 30 epochs.

C.2 Multi-label classification

The multi-label classification experiments use a shared ResNet-18 representation with separate task-specific classifiers for 40 binary attributes. Models are trained with cross-entropy losses and evaluated using attribute-wise and average test accuracy.

  • ResNet-18 without its final layer serves as the shared representation function for multi-label classification.
  • Forty separate 2048 × 2 fully connected layers provide task-specific functions for the 40 attributes.Each layer's two-dimensional output is passed through a 2-class softmax to produce binary attribute probabilities.
  • Cross-entropy is used as the task-specific loss for binary attribute classification.
  • Models use SGD with momentum, halve the learning rate every 30 epochs, train for 100 epochs, and use batch size 256.Learning rates are selected from LR = {1e−4, 5e−4, 1e−3, 5e−3, 1e−2, 5e−2} using validation accuracy.
  • Evaluation reports attribute-wise binary accuracies and average accuracy on the test set.CelebA images are resized to 64 × 64 × 3 before training.

C.3 Scene understanding

The scene understanding experiments use Cityscapes images resized to 256 × 512, with a fully convolutional ResNet-50 encoder and a pyramid pooling decoder. The section also references the architectures used for scene understanding and multi-label classification experiments.

  • Cityscapes images are resized to 256 × 512 for computational efficiency in the scene understanding experiments.
  • A fully convolutional ResNet-50, using layers before average pooling, serves as the shared scene-understanding encoder.
  • The scene-understanding decoder uses a pyramid pooling module.The supplied passage does not specify the decoder's output sizes beyond beginning that description.
  • The multi-label classification architecture is visualized in Figure 7.
  • Scene-understanding training uses SGD with momentum, halves the learning rate every 30 epochs, uses batch size 8, and runs for 250 epochs.
  • The scene understanding architecture is visualized in Figure 8.
Loading 1810.04650v2…