Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyond

1 INTRODUCTION

• Conflicting objectives arise in multi-task learning, fairness-aware learning, LLM alignment, and federated learning, where each task or criterion can be treated as an objective.Source: §1 · p. 2
• Pareto optimization represents trade-offs among objectives because improving one objective can worsen another, while user preference vectors specify objective importance.Source: §1 · p. 2
• Gradient-free population methods struggle with neural networks' millions or billions of parameters, where curse-of-dimensionality effects can cause slow or failed convergence.Source: §1 · p. 2
• Gradient-based multi-objective learning still faces difficult preference incorporation, high training cost, Pareto-set approximation cost, and mini-batch-induced optimization variance.Source: §1 · pp. 2–3

1.1 Comparison with Related Surveys on MOO

• Unlike earlier surveys, this survey focuses on gradient-based multi-objective deep learning and covers algorithms, theories, and applications across single, finite, and continuous solution sets.Source: §1.1 · p. 3

1.2 Organization of the Survey

• The taxonomy groups methods into balanced single-model approaches, finite Pareto-solution approaches, and continuous Pareto-set approaches that generate models for preferences on demand.Source: §1.2 · p. 4
• The survey additionally covers MOO preliminaries, convergence and generalization theory, applications, datasets, software tools, challenges, and future directions.Source: §1.2 · p. 4

2.1 Key Concepts in Multi-Objective Optimization

• Pareto optimal solutions cannot improve one objective without worsening another, while weakly Pareto optimal solutions exclude only strict dominance.Source: §2.1 · pp. 5–6
• The Pareto set contains Pareto-optimal decision solutions, whereas the Pareto front contains their objective-function values.Source: §2.1 · p. 6
• A preference vector lies on the simplex, with entries representing the importance assigned to individual objectives.Source: §2.1 · p. 6
• Pareto stationarity is necessary for Pareto optimality and becomes a KKT sufficient-and-necessary condition when objectives are convex and all preference weights are positive.Source: §2.1 · p. 6
• Hypervolume measures both convergence and diversity of a solution set, with larger values indicating solutions that are more diverse and closer to the Pareto front.Source: §2.1 · pp. 6–7

2.2 Scalarization Methods

• Linear scalarization is simple but can identify only convex Pareto-front regions and, for concave fronts, may find only endpoints.Source: §2.2 · p. 7
• Tchebycheff scalarization can cover both convex and concave Pareto-front regions across preference vectors, but its max operator causes nonsmoothness and a slow O(1/𝜖2) convergence rate.Source: §2.2 · p. 7

3 FINDING A SINGLE PARETO OPTIMAL SOLUTION

• Single-solution methods target one model that balances objectives, often treating them equally; equal weighting can produce unsatisfactory task performance, motivating dynamic objective-weight tuning.Representative approaches are categorized as loss balancing and gradient balancing methods.Source: §3 · p. 7

3.1 Loss Balancing Methods

• Loss balancing methods adjust objective weights using training-loss dynamics, uncertainty, loss scale, validation performance, or learned transformations; MOML uses validation-based bilevel optimization but incurs costly hypergradient computations.Auto-λ replaces MOML’s multi-objective upper-level problem with a single-objective approach and is therefore more efficient, while FORUM avoids hypergradients with a first-order method.Source: §3.1 · p. 8
• The logarithm transformation recovers IMTL-L’s transformation when its parameter is the exact per-iteration minimizer, while Smooth Tchebycheff scalarization improves smoothness and convergence while retaining convexity and Pareto optimality.Random Weighting has a higher probability of escaping local minima than Equal Weighting and consequently better generalization performance.Source: §3.1 · pp. 8–9

3.2 Gradient Balancing Methods

• Gradient balancing methods aggregate objective gradients into a common update direction and usually achieve better performance than loss balancing methods.Gradient weighting methods compute weighted gradient combinations, whereas gradient manipulation methods modify conflicting gradients before aggregation.Source: §3.2 · p. 9, §3.2.1 · p. 9, §3.2.2 · p. 11

3.2.1 Gradient Weighting Methods.

• Gradient weighting methods include MGDA, CAGrad, IMTL-Grad, Nash-MTL, FairGrad, Aligned-MTL, GradNorm, and DB-MTL, balancing descent, conflicts, projections, conditioning, or gradient magnitudes.MGDA and CAGrad can converge to a Pareto stationary point with full gradients, while stochastic-gradient convergence remains an active concern; DB-MTL enforces equal gradient norms each iteration, but its update magnitude affects performance.Source: §3.2.1 · p. 10

3.2.2 Gradient Manipulation Methods.

• Gradient manipulation methods such as PCGrad project conflicting gradients onto normal planes, removing components opposing other objectives; GradVac generalizes this correction adaptively.PCGrad applies correction only when gradient similarity is negative, whereas GradVac uses a more general adaptive formulation.Source: §3.2.2 · p. 11
• Gradient-manipulation methods address conflicts through exponentially averaged cosine similarities in GradVac, sign-consistent probabilistic masking in GradDrop [22], and related gradient adjustments.GradVac reduces to PCGrad when its averaged cosine similarity is zero.Source: §3.2.2 · p. 11

3.2.3 Practical Speedup Strategy.

• Speedup strategies reduce the cost of computing m objective gradients by using feature-level gradients, sampling objectives, delaying weight updates, or approximating MGDA weights from loss changes.Feature-level gradients have lower dimensionality than parameter-level gradients, whereas some approximations can degrade performance or apply only to MGDA-based methods.Source: §3.2.3 · p. 12

3.3 Discussions

• Although loss and gradient balancing can achieve a Pareto-optimal solution, they find only one solution without controlling its Pareto-front position; gradient balancing generally offers better convergence and can guarantee a Pareto stationary point.Speedup strategies may reduce computational and memory costs but often cause performance degradation.Source: §3.3 · p. 12
• Loss-balancing methods generally cost about the same as equal weighting because they use one backpropagation per iteration, while gradient balancing requires m backpropagations and is substantially more expensive.Exceptions among loss-balancing methods include MOML [212], Auto-λ [114], and FORUM [211].Source: §3.3 · p. 12

4 FINDING A FINITE SET OF SOLUTIONS

• Finite-solution methods provide a discrete approximation of the Pareto set so users can select among multiple trade-offs.Source: §4 · pp. 12–13

4.1 Methods Based on Preference Vectors

• Preference-vector methods solve subproblems associated with predefined vectors, but PMTL controls sectors rather than precisely positioning Pareto solutions.Source: §4.1 · p. 13
• EPO aligns solutions exactly with preference vectors through a uniformity function, while its drawbacks include non-negative objectives, complex set partitioning, and repeated Jacobian and linear-programming computations.Source: §4.1 · pp. 13–14
• Preference-vector approaches include theoretical distribution guarantees from UMOD, adaptive preference optimization in GMOOAR, and convergence rate O(𝜖−1) for FERERO with deterministic gradients.Source: §4.1 · p. 14

4.2 Methods without Using Preference Vectors

• Methods without preference vectors directly optimize diverse solution sets, including hypervolume-gradient, dimension-sweeping, MOO-SVGD, and objective-to-solution assignment approaches.Source: §4.2 · pp. 14–15

4.3 Discussions

• Preference-vector methods enable independent subproblems, memory efficiency, and parallel computation, whereas direct multi-solution methods avoid preference selection but can share information across solutions.Source: §4.3 · pp. 15–16

5 FINDING AN INFINITE SET OF SOLUTIONS

• The section motivates infinite Pareto-set learning because finite solution sets only discretely approximate the front and cannot represent solutions for every user preference.Learning infinitely many solutions individually is impractical, motivating preference-to-solution mappings.Source: §5 · p. 16
• First-order expansion around discrete Pareto-optimal solutions suffers increasing error for widely spaced solutions, poor high-dimensional performance, and computationally difficult linear-program updates.Neural mappings address these issues by directly learning preference-to-solution functions.Source: §5 · p. 16

5.1.1 Hypernetwork.

• CPMTL [100] uses user preference α as input to a hypernetwork whose output is the target, while a hypernetwork architecture demonstrates superior performance over MLP-based hypernetwork models.Hypernetworks can be much larger than base networks because of target-network parameter counts, limiting applicability to large models; chunking addresses this limitation while preserving accurate parameter production.Source: §5.1.1 · p. 16

5.1.2 Preference-Conditioned Network.

• Preference-conditioned networks inject preferences into the model, using FiLM to modulate features and enabling stronger conditioning than input concatenation.Dynamic architecture prediction further adapts model structure to user preferences, although simple input conditioning may produce insufficiently diverse solutions.Source: §5.1.2 · pp. 16–17

5.1.3 Model Combination.

• Model-combination methods can introduce diversity and provide users with broader choices; PaMaL [40] learns several individual models and combines their parameters through a weighted sum based on user preferences.LORPMAN [19] uses full-rank and low-rank components to manage up to 40 objectives, achieving superior performance.Source: §5.1.3 · p. 17

5.2 Training Strategy

• Training samples user preferences and optimizes structural parameters so generated networks align with those preferences.Existing methods use scalarization, preference-aware weighting, or hypervolume maximization; PHN-HVI [62] samples preference vectors and promotes both diversity and convergence.Source: §5.2 · p. 18

5.3 Discussions

• Most algorithms favor linear scalarization for simplicity and performance, while preference-dependent weighting is important when aligning solutions with user preferences.Structural choice and optimization strategy jointly determine the effectiveness of infinite Pareto-set learning.Source: §5.3 · pp. 18–19
• Hypernetworks offer flexibility but are unsuitable for some large models, whereas preference-conditioned networks add few parameters but may approximate complex Pareto sets less accurately.Model combinations provide an intermediate option by balancing Pareto-set approximation, scalability, and adjustable low-rank parameterization.Source: §5.3 · pp. 18–19
• HV-based optimization criteria in Section 4.2 can achieve HV-optimal solutions by focusing on dominated objective-space volume to better capture the Pareto front.Source: §5.3 · p. 19

6.1.1 Deterministic Gradient.

• O(K^-1/2) convergence to a Pareto-stationary point is established for MGDA under deterministic gradients, matching single-objective optimization rates.MGDA's common descent direction improves every objective by at least O(∥d∥^2) when the iterate is not Pareto-stationary.Source: §6.1.1 · p. 19

6.1.2 Stochastic Gradient.

• Stochastic-gradient analyses address bias and variance through smoothing, delayed weight updates, tracking variables, and double sampling.Smoothing stabilizes common descent directions, while MoDo obtains unbiased gradient-product estimates without bounded function values.Source: §6.1.2 · pp. 19–20
• O(ε^-2) sample complexity matches stochastic single-objective optimization for delayed-weight methods, while MoDo provides convergence guarantees and bounded conflict-avoidant distance.These results are summarized in Table 2, whose columns encode smoothness, boundedness assumptions, sample complexity, and conflict-avoidant distance.Source: §6.1.2 · p. 20, Table 2 · p. 20
• Stochastic convergence remains practically constrained by large iteration-dependent batch sizes, many-objective scaling, and bounded-function-value assumptions.Some analyses require batch sizes growing linearly with iterations, increasing computational and memory costs; MoCo also becomes less practical as the number of objectives grows.Source: §6.1.2 · pp. 19–20

6.2 Generalization

• Generalization theory uses sample-complexity, regret, stability, and Rademacher-complexity analyses, but online-to-batch methods still lag optimal offline complexity.Scalarization results extend individual-objective bounds to MOO, while MoDo derives algorithm-dependent bounds through stability.Source: §6.2 · p. 21

7.1 Reinforcement Learning

• MORL uses scalarization, preference-vector configuration, gradient balancing, and preference-conditioned policies to approximate or learn Pareto-optimal policies.Uniform preference sampling need not yield uniformly distributed solutions, motivating adaptive methods such as OLS and PGMORL.Source: §7.1 · p. 22

7.2 Bayesian Optimization

• MOBO balances exploration and exploitation through scalarization, acquisition-function extensions, hypervolume improvement, and information-theoretic candidate selection.Most methods produce discrete Pareto-set approximations, while hypernetworks and later generative approaches target continuous approximations.Source: §7.2 · p. 22

7.3 Computer Vision

• Computer-vision multi-task learning shares encoders to reduce computation but can create task conflicts and performance drops, motivating MOO-based loss balancing.Applications include dense prediction, point-cloud processing, medical-image denoising, and pose estimation.Source: §7.3 · p. 23

7.4 Neural Architecture Search

• Neural architecture search treats accuracy and resource objectives such as parameters, FLOPs, and latency as a multi-objective problem.Hypernetworks map preference vectors to architectures, avoiding a complete search rerun when preferences change.Source: §7.4 · p. 23

7.5 Recommendation Systems

• Recommendation systems optimize quality dimensions including novelty, diversity, serendipity, and popularity using scalarization, gradient balancing, and finite or infinite Pareto sets.Finite and infinite Pareto-set methods support personalized recommender models for varying user preferences.Source: §7.5 · p. 23

7.6 Large Language Models (LLMs)

• LLM multi-objective training addresses task adaptation and alignment across preferences, but training multiple models creates substantial computational and storage costs.Preference-aware methods condition prompts or parameters, while MFTCoder and CoBa address task imbalance, difficulty, convergence, and overhead.Source: §7.6 · pp. 23–24

7.7 Miscellaneous

• MOO has also been applied to meta-learning, federated learning, long-tailed learning, continual learning, diffusion models, GFlowNets, and PINNs.These applications illustrate the breadth of multi-objective trade-offs in deep learning.Source: §7.7 · p. 24

8.1 Datasets

• Computer-vision benchmarks include NYUv2, Taskonomy, and Cityscapes, covering dense prediction tasks with reported training and testing sets.Taskonomy contains 26 tasks and about 4 million samples, whereas Cityscapes contains two tasks with 2,975 training and 500 testing samples.Source: §8.1 · pp. 24–25
• The collection includes QM9 molecular property prediction with 11 regression tasks and 130,000 samples, CelebA with 40 binary face-attribute tasks, and Meta-World’s MT10/MT50 robot-manipulation settings containing 10 and 50 tasks, respectively.Source: §8.1 · p. 25
• LLM-oriented resources include XTREME, evaluating cross-lingual generalization across 9 tasks in 40 languages; PKU-SafeRLHF, with 82,118 question-answering pairs annotated for helpfulness and harmlessness; and UltraFeedback, containing 63,967 prompts, each paired with four responses.Source: §8.1 · p. 25

8.2 Libraries

• LibMTL and LibMOON provide modular PyTorch environments for fair evaluation and method development.LibMTL focuses on single Pareto-optimal solutions with 24 methods and six datasets, whereas LibMOON covers finite and infinite Pareto sets with over 20 methods.Source: §8.2 · p. 25

Theoretical Understanding

• Theory has emphasized convergence to stationary points more than generalization on unseen data, leaving broader algorithm-dependent and Pareto-set analyses open.Extending generalization analysis beyond existing examples could clarify how different techniques affect unseen-data performance.Source: §9 · p. 26

Pareto-set learning

• Pareto-set learning still lacks clear evidence about how architectures and training strategies affect approximation quality.Theoretical understanding of Pareto-set approximation remains limited alongside broader gaps in generalization analysis.Source: §9 · p. 26

Reducing Gradient Balancing Costs

• Gradient-balancing methods retain substantial computational overhead, motivating analyses that could combine them with scalarization or loss balancing.Such integration is proposed as a route toward large-scale training.Source: §9 · p. 26

Dealing with Large Number of Objectives

• Many-objective Pareto-set learning faces exponentially expanding preference spaces, while objective reduction or merging may reduce the total burden.More efficient sampling strategies are needed for high-dimensional preference spaces.Source: §9 · p. 26

Distributed Training

• Scaling MOO to distributed environments requires efficient gradient communication, multi-model synchronization, and collaboration without direct data sharing.These requirements become especially important when objective-specific data cannot be shared because of privacy constraints.Source: §9 · p. 26

Advancements in Large Language Models (LLMs)

• LLM MOO research largely focuses on RLHF, leaving pre-training, inference, and richer representations of complex user preferences for future work.Preference vectors may oversimplify user preferences, motivating more advanced representations for personalization.Source: §9 · pp. 26–27

Application in More Deep Learning Scenarios

• Further applications remain open because deep-learning systems commonly involve multiple evaluation criteria and naturally occurring trade-offs.The survey frames broader use of MOO as an opportunity across additional deep-learning scenarios.Source: §9 · p. 27

• By unifying algorithms, theories, applications, and challenges, the paper provides a foundational resource for future advancements in the field.Source: §10 · p. 27
• The survey systematically covers algorithms producing a single balanced model, a finite set of models, and an entire infinite Pareto set.Source: §10 · p. 27
• It reviews theoretical foundations, including convergence results, while highlighting the need for deeper understanding of generalization.Source: §10 · p. 27
• The review demonstrates multi-objective deep learning's practical significance across diverse applications, including emerging applications in Large Language Models (LLMs).Source: §10 · p. 27