PaperlayerView exact revision on arXiv

Source-linked AI summary

Conflict-Averse Gradient Descent for Multi-task Learning

Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, Qiang Liu

arXiv:2110.14048v2cs.LGcs.AI

TL;DR

Multi-task learning can suffer when conflicting task gradients make average-loss optimization detrimental to individual tasks, while prior methods may lack convergence guarantees or select arbitrary Pareto-stationary points. CAGrad regularizes updates using the worst local task improvement, balancing objectives while provably converging to an average-loss minimum. It generalizes GD and MGDA and improves performance across challenging multi-task problems.

  • Problem

    Conflicting task gradients can make average-loss optimization detrimental to individual-task performance, while prior gradient-manipulation methods may lack convergence guarantees or converge to arbitrary Pareto-set points.

  • Method

    CAGrad selects an update near the average gradient that maximizes the worst local improvement among task objectives, with a hyper-parameter controlling the convergence rate.

  • Results

    CAGrad improves performance over prior state-of-the-art gradient-manipulation methods across challenging supervised, semi-supervised, and reinforcement-learning multi-task problems.

  • Takeaways & Limitations

    CAGrad targets the average-loss optimum while reducing gradient conflict and generalizes gradient descent and MGDA as special cases.

  • Takeaways & Limitations

    The convergence guarantee assumes differentiable losses with H-Lipschitz gradients and a fixed step size α satisfying 0 < α ≤ 1/H.

Abstract

from arXiv · show

The goal of multi-task learning is to enable more efficient learning than single task learning by sharing model structures for a diverse set of tasks. A standard multi-task learning objective is to minimize the average loss across all tasks. While straightforward, using this objective often results in much worse final performance for each task than learning them independently. A major challenge in optimizing a multi-task model is the conflicting gradients, where gradients of different task objectives are not well aligned so that following the average gradient direction can be detrimental to specific tasks' performance. Previous work has proposed several heuristics to manipulate the task gradients for mitigating this problem. But most of them lack convergence guarantee and/or could converge to any Pareto-stationary point. In this paper, we introduce Conflict-Averse Gradient descent (CAGrad) which minimizes the average loss function, while leveraging the worst local improvement of individual tasks to regularize the algorithm trajectory. CAGrad balances the objectives automatically and still provably converges to a minimum over the average loss. It includes the regular gradient descent (GD) and the multiple gradient descent algorithm (MGDA) in the multi-objective optimization (MOO) literature as special cases. On a series of challenging multi-task supervised learning and reinforcement learning tasks, CAGrad achieves improved performance over prior state-of-the-art multi-objective gradient manipulation methods.

1 Introduction

Multi-task learning shares one model across tasks for efficiency, but directly optimizing average loss can harm individual-task performance because gradients conflict. CAGrad addresses this by reducing gradient conflict while converging to an average-loss minimum.

  • Multi-task learning shares parameters across tasks, enabling more efficient learning with a smaller overall model than separate models.
  • Conflicting gradients can differ in scale or direction, allowing one task to dominate updates or making average-loss optimization detrimental to another task.
  • Existing gradient-manipulation methods may lack convergence guarantees or converge to arbitrary Pareto-set points depending on initialization.
  • CAGrad maximizes the worst local improvement within a neighborhood of the average gradient, automatically balancing objectives while converging to an average-loss minimum.
  • CAGrad includes gradient descent and MGDA as special cases and improves over prior gradient-manipulation methods on challenging supervised, semi-supervised, and reinforcement-learning problems.

2 Background

Multi-task learning optimizes several task losses for one shared parameter set, commonly through their average. Conflicting task gradients can make this natural update direction degrade individual-task optimization, motivating gradient-manipulation approaches.

  • Multi-task learning uses shared parameters θ across K ≥2 tasks, each with a loss function, to achieve low losses across tasks.
  • The standard multi-task objective minimizes the average loss over all task losses.
  • Direct gradient descent on the average loss may significantly compromise optimization of individual losses because of conflicting gradients.
  • An averaged gradient g0 can conflict with an individual gradient gi when ⟨gi, g0⟩ < 0, so following g0 decreases that task’s performance.
  • MGDA targets the Pareto set, heuristic methods re-weight task losses, and PCGrad projects task gradients to reduce conflict.

3 Method

CAGrad chooses updates that reduce the average loss while maximizing the worst local improvement across tasks. It retains convergence to a stationary point of the average loss for 0 ≤ c < 1, supports task subsampling, and includes GD and MGDA as limiting cases.

  • 3.1 Conflict-Averse Gradient Descent: CAGrad selects an update within a ball around the average gradient that minimizes measured conflict among task losses.The hyperparameter c controls the ball and convergence rate, while the conflict measure is based on the minimum decrease rate across losses.
  • 3.1 Conflict-Averse Gradient Descent: For c = 0, CAGrad recovers gradient descent; as c →∞, it approaches MGDA’s minimum-norm convex-hull update.MGDA leaves all Pareto-set points as fixed points and therefore does not control which Pareto point is reached.
  • 3.2 Convergence Analysis: For 0 ≤ c < 1, CAGrad converges to a stationary point of the average loss gradient ∇L0.The theorem assumes differentiable losses with H-Lipschitz gradients and a fixed step size satisfying 0 < α ≤ 1/H.
  • 3.2 Convergence Analysis: CAGrad’s update-vector illustration contrasts GD, MGDA, PCGrad, and CAGrad for two task gradients g1 and g2.PCGrad projects each gradient onto the other’s normal plane, whereas CAGrad searches within a ball around the average gradient.
  • 3.3 Practical Speedup: Task subsampling reduces computation by optimizing with gradients from a subset S while retaining the convergence guarantee.The stated complexity is O((|S|N + T), and choosing S = [K] recovers full CAGrad.

4 Related Work

Related work addresses multi-task learning through architecture design and optimization methods. CAGrad is presented as model-agnostic and as improving theoretical convergence and empirical performance over prior approaches.

  • Multi-task Learning: MTL architectures use task-specific modules, attention mechanisms, or different network paths to address multiple tasks.These approaches accompany MTL’s broader benefits in data and computational efficiency across vision, language, and robotics.
  • MTL Optimization: MTL optimization methods re-weight task losses or manipulate gradients to balance learning across objectives.Re-weighting criteria include uncertainty, gradient norm, and task difficulty, while MGDA and PCGrad seek better local update directions.
  • MTL Optimization: CAGrad is model-agnostic, converges to the optimal point in theory, and achieves better empirical performance on toy and real-world tasks.The comparison is made against prior MTL optimization techniques within the paper’s supported experimental scope.

5 Experiment

Experiments test CAGrad’s convergence behavior, recovery of GD and MGDA through c, and performance across supervised, reinforcement, and semi-supervised multi-task settings. CAGrad achieves favorable results while balancing task objectives across these evaluations.

  • 5.1 Convergence and Ablation over c: CAGrad, MGDA, and PCGrad are evaluated on a toy optimization example, while c is varied to test CAGrad’s recovery of GD and MGDA.The experiments use five initial parameter vectors and compare optimization trajectories.
  • 5.1 Convergence and Ablation over c: CAGrad achieves the lowest average loss on Multi-Fashion+MNIST with c = 0.2.With c = 0 and c = 10, CAGrad roughly recovers GD and MGDA performance, respectively.
  • 5.2 Multi-task Supervised Learning: On NYU-v2 and CityScapes, CAGrad improves the task overlooked by other methods and matches their performance on the remaining tasks.The reported improvements concern Surface Normal on NYU-v2 and Depth on CityScapes.
  • 5.3 Multi-task Reinforcement Learning: On Meta-World MT10 and MT50, CAGrad outperforms all baselines except CARE, which benefits from extra metadata information.CAGrad-Fast uses task subsampling and achieves comparable performance against the state-of-the-art method.
  • 5.4 Semi-supervised Learning with Auxiliary Tasks: Across CIFAR10 experiments with 500, 1000, and 2000 labeled images, CAGrad yields the best averaged test accuracy.MGDA performs much worse than the ARML baseline because it significantly overlooks the main classification task.

6 Conclusion

CAGrad optimizes the minimum decrease rate across task losses while converging to the average-loss optimum. It generalizes GD and MGDA and improves performance across challenging multi-task problems.

  • CAGrad explicitly optimizes the minimum decrease rate of any task loss while provably converging to the optimum of the average loss.
  • CAGrad generalizes gradient descent and MGDA, and improves performance across several challenging multi-task learning problems.

Checklist

The checklist reports reproducibility details, asset disclosures, theoretical documentation, and a scope limitation concerning the choice of main objective.

  • The paper mainly focuses on optimizing the average loss, which could be replaced by other main objectives.
  • The theoretical assumptions and complete proofs are stated in Theorem 3.2 and Appendix A.3.
  • The paper reports that experiment details, training settings, code, and data are provided through the main text, appendices, or supplementary material.
  • Experiments use multiple seeds, and computational efficiency and resource details are reported.
  • The work uses cited publicly available assets under MIT licenses and introduces no new assets beyond a modified shrinked LeNet.

A.2 Projecting Conflicting Gradients (PCGrad)

PCGrad projects task gradients to remove pairwise conflicts before combining them, but its optimization objective and convergence behavior are limited.

  • PCGrad projects each task gradient onto the normal plane of other gradients before combining them into an update vector.
  • PCGrad lacks a clear per-step optimization objective, making general convergence analysis difficult.
  • PCGrad is guaranteed to converge to a Pareto-stationary point without explicit control over which point it reaches.
  • PCGrad’s convergence guarantee applies only to two-task learning under convexity, differentiability, and a Lipschitz-gradient condition.

Convergence Analysis

Under smoothness and step-size assumptions, CAGrad has convergence guarantees for the average loss and Pareto stationarity, with behavior depending on its parameter regime and step-size schedule.

  • Under Assumption A.3 and fixed 0 < α ≤ 1/H, CAGrad converges to stationary points of L0 when 0 ≤ c < 1.
  • For any c ≥ 0, every fixed point of CAGrad is Pareto-stationary for the objectives (L0, L1, . . . , LK).
  • When c ≥ 1 with a properly decaying step size, CAGrad limit points are either stationary points of L0 or Pareto-stationary points of the individual task losses.
  • For c > 1, the convergence behavior depends on the specific choice of step size.

B.2 Multi-task Supervised Learning

The supervised-learning experiments evaluate CAGrad on NYU-v2 and CityScapes under a shared MTAN-based setup, including test loss, training time, and ablations.

  • Experimental setup: NYU-v2 and CityScapes experiments use the MTAN setup with SegNet as the backbone and MTAN attention.CityScapes uses 7-class semantic labels.
  • Evaluation: Test loss and training time are compared for the supervised-learning methods on NYU-v2 and CityScapes.The comparison is reported in Figure 5.
  • Ablations: Additional studies examine uncertain weighting, different CAGrad c values, and a doubled PCGrad learning rate.The reported results include Tables 5 and 6.

B.3 Multi-task Reinforcement Learning

The reinforcement-learning experiments test CAGrad on MetaWorld’s MT10 and MT50 benchmarks, including an efficiency comparison with PCGrad and CAGrad-Fast.

  • Experimental setup: The reinforcement-learning experiments use the MetaWorld MT10 and MT50 benchmarks with Soft Actor Critic as the underlying algorithm.MT10 is a 10-task subset of the MT50 task pool.
  • Benchmarks: MT50 contains 50 tasks, while MT10 contains a subset of 10 tasks from the same task pool.The task sets are visualized in Figure 6.
  • Optimization: CAGrad’s objective is approximately solved with 20 gradient-descent steps because reinforcement learning involves more tasks than supervised MTL.The gradient-descent learning rates are 25 for MT10 and 50 for MT50.
  • Results: CAGrad-Fast achieves comparable or better results than PCGrad with roughly 2x and 5x speedups on MT10 and MT50, respectively.The comparison concerns computational efficiency and task performance.
  • Efficiency: PCGrad’s sequential random task ordering requires a loop that makes it slow for large task counts in practice.Its theoretical time complexity is described as the same as CAGrad’s.

B.4 Semi-Supervised Learning with Auxiliary Tasks

The semi-supervised auxiliary-task experiments compare training losses across methods and show that gradient manipulation affects both task balance and averaged loss.

  • Evaluation: Training losses are analyzed at the last epoch when 2,000 labeled images are used.Table 8 reports averaged losses over three independent runs.
  • Training-loss analysis: MGDA achieves the smallest loss on the second auxiliary task while totally ignoring the main task loss LCE.This behavior is interpreted as finding a sub-optimal solution on the Pareto front.
  • Training-loss analysis: CAGrad and PCGrad both reduce the averaged loss L0 relative to the ARML baseline, with CAGrad producing the smaller L0.The comparison is reported for the semi-supervised auxiliary-task setting.
  • Reporting: The reported losses replace values smaller than 10^-6 with 0 and mark the smallest losses in bold.Each method’s values are averaged over three independent runs.
Loading 2110.14048v2…