PaperlayerView exact revision on arXiv

Source-linked AI summary

SIMS: Scale-Invariant Merit-Function-Based Scalarization for Multi-Task Learning

Zebin Chen, Fei Xing, Yang Chen, Hua Liu, Andy HF Chow, Yuhua Qian, Yu Zhang

arXiv:2609.12599v1cs.LGcs.AImath.OC

TL;DR

Existing merit-function scalarization can be sensitive to task-loss scales even though rescaling leaves Pareto-optimal solutions unchanged. SIMS uses a logarithmic transformation-induced merit function with weak Pareto guarantees and a smooth surrogate, and it achieves state-of-the-art performance across three benchmarks.

  • Problem

    Existing merit-function-based scalarization methods are sensitive to task-loss scales, although positive rescaling does not change Pareto-optimal solutions.

  • Method

    SIMS uses a logarithmic transformation-induced merit function that is scale-invariant, preserves weak Pareto optimality, and admits a smooth surrogate with controllable approximation error.

  • Results

    SIMS achieves state-of-the-art average performance across NYUv2, Cityscapes, and PASCAL-Context.

  • Takeaways & Limitations

    SIMS provides a scale-invariant scalarization approach for converting MTL multi-objective optimization into a single objective while retaining a weak Pareto characterization.

  • Takeaways & Limitations

    The exact merit function is generally nonsmooth, preventing direct gradient descent and necessitating slower-converging subgradient-based methods.

Abstract

from arXiv · show

Multi-task learning (MTL) requires navigating unavoidable trade-offs among competing objectives. This paradigm is frequently formulated as multi-objective optimization (MOO), where the scalarization is favored to reduce an MOO problem to a single objective. We empirically find that existing merit-function-based scalarization approaches are sensitive to the relative scales of different objectives in practical MTL, where task losses commonly differ by orders of magnitude. The optimization process often favors objectives with larger scales even though the underlying Pareto optimal solutions remains invariant to rescaling (i.e., multiplying an objective by a positive constant). To address this issue, we propose Scale-Invariant Merit-function-based Scalarization (SIMS) for MTL. Specifically, SIMS adopts a transformation-induced merit function to convert the MOO problem of MTL to a single objective that renders optimization invariant to the magnitudes of losses. Theoretically, we prove that the requirement for scale invariance uniquely determines this transformation to be logarithmic. We further show that this general transformation-induced merit function preserves weak Pareto optimality and admits a smooth surrogate with controllable approximation error. Extensive experiments on representative multi-task benchmarks demonstrate that SIMS consistently outperforms existing scalarization methods and achieves state-of-the-art performance.

1 Introduction

MTL faces competing task objectives, motivating multi-objective scalarization to seek Pareto-optimal solutions without manually tuned weights. This paper proposes SIMS to make merit-function scalarization scale-invariant while preserving weak Pareto optimality and achieving strong benchmark performance.

  • 1 Introduction: MTL uses shared representations across related tasks but must navigate unavoidable trade-offs because tasks are rarely perfectly aligned.The paradigm is used in applications including dense prediction, autonomous driving, and recommendation systems.
  • 1 Introduction: Existing merit-function scalarization methods can be substantially affected by task-loss rescaling even though Pareto-optimal solutions remain unchanged.Task losses in practical MTL may differ by orders of magnitude, making relative scale a consequential optimization issue.
  • 1 Introduction: SIMS applies a transformation-induced merit function that is intrinsically invariant to task-loss scales while preserving a principled characterization of weak Pareto optimality.The method theoretically identifies logarithmic transformations as necessary for scale invariance.
  • 1 Introduction: SIMS admits theoretical analysis establishing its logarithmic transformation, weak Pareto characterization, and smooth surrogate with controllable approximation error.These properties connect scale invariance with both Pareto theory and practical optimization.
  • 1 Introduction: Experiments on multi-task benchmarks show that SIMS consistently outperforms existing scalarization approaches and achieves state-of-the-art performance.The contribution summary specifically reports extensive benchmark evidence for effectiveness.

2 Related work

Related MTL work addresses task conflicts through adaptive gradient methods and scalarization. Scalarization is attractive for reducing multiple objectives to one, but linear forms can miss non-convex Pareto-front solutions.

  • 2 Related work: Adaptive gradient methods construct task-balancing descent directions, with later approaches targeting stability or fairness through gradient aggregation.Examples include MGDA, CAGrad, PCGrad, Nash-MTL, and FairGrad.
  • 2 Related work: Scalarization converts multiple objectives into a single scalar objective for optimization.This formulation underlies substantial prior MTL work.
  • 2 Related work: Linear scalarization methods combine task losses with weights, including equal weighting, uncertainty weighting, dynamic weight averaging, and GradNorm.These methods represent a major prior scalarization paradigm in MTL.
  • 2 Related work: Linear scalarization can miss solutions on the non-convex part of the Pareto front.This limitation motivates considering alternative scalarization approaches.

3 Preliminary

The paper formulates MTL as multi-objective optimization over task losses and uses merit functions to connect Pareto analysis with single-objective optimization. It defines Pareto notions and introduces maximum-loss and Chebyshev merit functions.

  • 3.1 MTL as MOO: Each task loss is treated as an objective over shared model parameters in the MTL multi-objective formulation.The formulation contains m related tasks, with each L_i(θ) mapping parameters to a nonnegative loss.
  • 3.1 MTL as MOO: A weakly Pareto optimal solution has no alternative that strictly improves every task loss simultaneously.This is weaker than Pareto optimality, which permits no alternative that weakly improves all objectives and strictly improves one.
  • 3.2 Merit Functions: Merit functions provide nonnegative surrogates whose zeros characterize weakly Pareto optimal solutions.They connect multi-objective optimization with single-objective minimization.
  • 3.2 Merit Functions: The maximum-loss merit function uses the largest normalized task loss relative to each objective’s minimal attainable value.The Chebyshev merit function extends this construction with weights that emphasize different objectives.
  • 3.2 Merit Functions: Stationarity measures whether the gradient norm of a differentiable merit function is at most a specified ε.When ε=0, the point is stationary.

4 Motivating Examples

Task-loss scales commonly differ because of heterogeneous tasks and implementation choices, although positive rescaling leaves Pareto solutions unchanged. Synthetic experiments show that several scalarization methods are scale-sensitive, while SIMS is more consistent across scalings.

  • 4 Motivating Examples: Task-loss scales differ across MTL tasks because of heterogeneous loss functions, task types, reduction conventions, and task-specific normalization.Examples include regression versus classification and sum versus mean aggregation.
  • 4 Motivating Examples: Rescaling changes gradient magnitudes and can redirect scalarization optimization toward extreme Pareto trade-offs that substantially sacrifice some tasks.Objectives with larger scales can dominate the optimization procedure.
  • 4 Motivating Examples: EW, STCH, and FOOPS follow different optimization trajectories under 10:1 and 1:10 task-loss scalings despite an unchanged Pareto front.All three methods drift toward the task with the larger scale.
  • 4 Motivating Examples: FOOPS can become unstable and fail to converge to a stationary solution, while GLS remains scale-invariant but may converge away from the Pareto front because of local optima.The example indicates that scale insensitivity alone does not ensure a valid merit-function criterion.
  • 4 Motivating Examples: SIMS exhibits scale-invariant behavior and more consistent convergence under different task-loss scalings.Its behavior is attributed to the proposed transformation-induced merit function.

5 Methodology

SIMS constructs a transformation-induced merit function whose logarithmic transformation makes scalarization invariant to task-loss scales while preserving weak Pareto optimality. A smooth, regularized surrogate provides controllable approximation error and supports stable optimization with convergence guarantees.

  • 5.1 Formulation: SIMS applies a shared monotonically increasing transformation to each task loss, then uses the resulting transformation-induced merit function for scalarization.Monotonicity preserves the ordering of each task’s solutions, while the merit function measures the largest simultaneous improvement achievable over the current point.
  • 5.2 Properties: The induced merit function is nonnegative and equals zero exactly at weakly Pareto-optimal solutions.Thus, minimizing the merit function preserves the desired weak Pareto characterization.
  • 5.1 Formulation: Scale invariance uniquely requires the transformation to have the logarithmic form ψ(x) = a ln x + b, with ψ(x) = ln x used in experiments.The constants a > 0 and b only positively affinely rescale objective values and therefore do not change the solution set.
  • 5.3 Approximation: SIMS replaces the generally nonsmooth merit function with a log-sum-exp-based smooth surrogate whose approximation error is controlled by τ.For λ = 0, the surrogate has error at most ε = τ ln m; its gradient is Lipschitz with constant L_v0,τ = C + α/τ.
  • 5.3 Approximation: The surrogate satisfies v0,τ(θ) ≤ ū(θ) ≤ v0,τ(θ) + τ ln m, giving a uniform approximation gap that tightens as τ decreases.When λ = 0, minimizing the surrogate yields an ε-weakly Pareto-optimal solution with ε = τ ln m.
  • 5.4 Algorithm and Convergence Analysis: When λ > μ̄, the inner problem is strictly convex with a unique minimizer, enabling stable gradient-based updates and a convergence guarantee for TTGDA.Theorem 5.9 bounds the iteration and gradient complexity required to obtain an ε-stationary point under stated assumptions and step sizes.

6 Experiments

Experiments evaluate SIMS across three multi-task benchmarks, hyper-parameter settings, transformation functions, normalization strategies, and scale-rescaling conditions. SIMS achieves state-of-the-art average performance, balanced task trade-offs, and substantially stronger scale-invariant behavior than existing methods.

  • Main Results: SIMS achieves state-of-the-art average performance across NYUv2, Cityscapes, and PASCAL-Context.Results are reported in Tables 1–3 across benchmarks with diverse task compositions and objective scales.
  • Main Results: SIMS maintains balanced task performance, whereas FOOPS can neglect depth estimation on Cityscapes because larger-scale segmentation losses dominate.EW consistently has the lowest performance among evaluated scalarization methods.
  • Main Results: SIMS maintains consistent optimization trajectories under objective scaling, avoiding extreme trade-off solutions observed in baseline methods.The scale-invariance behavior is corroborated by visualization of optimization trajectories.
  • Ablation Studies: SIMS performance remains stable for smoothing temperatures τ in [1] [10], achieving competitive or state-of-the-art overall performance.All other hyper-parameters are fixed to the settings used in the preceding experiments.
  • Ablation Studies: SIMS is insensitive to λ across several orders of magnitude, while small positive values sometimes provide modest gains; experiments therefore use λ = 0 uniformly.λ controls proximal regularization, affecting inner-problem conditioning and the surrogate’s Pareto interpretation.
  • Comparison with Loss Normalization: The logarithmic transformation outperforms alternative power and asinh transformations, while statistic-driven loss normalization underperforms SIMS by Δb of −1.20%, −0.38%, and −1.15%.The asinh setting c = 0.01 trails the logarithmic baseline by only Δb = −0.07%; expansive power scaling worsens loss-scale discrepancies.
  • Synthetic Experiments for Scale Invariance: Across seven rescaling settings, trajectory variances for EW, FOOPS, and STCH increase rapidly, whereas GLS and SIMS remain close to zero.Smaller variance indicates stronger scale-invariant behavior, assessed from both trajectories and final solutions.

7 Conclusion

SIMS addresses scale sensitivity in merit-function scalarization by using a logarithmic transformation that preserves weak Pareto optimality and supports smooth optimization. The paper establishes the transformation theoretically and demonstrates practical effectiveness on standard benchmarks.

  • 7 Conclusion: SIMS uses a logarithmic transformation-induced merit function to make scalarized optimization insensitive to task-loss magnitudes.Scale invariance uniquely determines the transformation as logarithmic.
  • 7 Conclusion: The transformation-induced merit function preserves weak Pareto optimality while converting the multiobjective problem into a single objective.The merit function is nonnegative, and its zero-value characterization supports weak Pareto optimality under the stated conditions.
  • 7 Conclusion: The formulation admits a smooth surrogate with controllable approximation error and established smoothness properties.The analysis includes log-sum-exp smoothing and Lipschitz-gradient bounds.
  • 7 Conclusion: The surrogate objective can be efficiently solved with TTGDA, which has established convergence guarantees.This algorithmic result connects the theoretical surrogate to an implementable optimization procedure.
  • 7 Conclusion: Extensive experiments on standard multi-task benchmarks demonstrate the effectiveness of SIMS.The supplied conclusion states benchmark effectiveness but does not provide specific quantitative values.

A.4 Proofs in Section 5.3

The appendix establishes properties of the smoothed merit function and its optimization analysis, including smoothness, weak convexity, approximation behavior, and convergence-related bounds. It also documents the illustrative setup and selected ablation evaluations.

  • A.4 Proofs in Section 5.3: Log-sum-exp smoothing yields a continuously differentiable function with Lipschitz gradient under Lipschitz and smooth task transformations.The resulting smoothness bound depends on the maximum task Lipschitz and smoothness constants.
  • A.4 Proofs in Section 5.3: The smoothed merit function converges uniformly to the original merit function as the smoothing parameter approaches zero.For λ=0, the limiting zero condition characterizes weak Pareto optimality.
  • A.4 Proofs in Section 5.3: Strict convexity in the inner variable ensures a unique minimizer for the smoothed inner problem.The proof combines weak convexity of the transformed objectives with the quadratic regularizer.
  • A.4 Proofs in Section 5.3: Under convexity of all transformed task losses, the surrogate condition implies weak Pareto optimality.The argument rules out a point that strictly improves every transformed objective.
  • A.4 Proofs in Section 5.3: The illustrative experiment uses five initialization points, Adam with learning rate 10^-3, and an 800 × 800 grid to approximate the Pareto front.Optimization runs for up to 35K iterations and stops early when distance to the approximated Pareto set is below 0.05.

B.2 Implementation Details

The implementation uses task-specific losses suited to segmentation, depth, surface-normal, and edge-pixel prediction. These choices account for valid annotations and the extreme sparsity of edge pixels.

  • B.2 Implementation Details: Segmentation tasks use pixelwise cross-entropy over valid pixels, ignoring the void class with index 255.The loss is applied over the image domain and excludes invalid labels.
  • B.2 Implementation Details: Depth estimation uses L1 loss, or mean absolute error, restricted to pixels with valid depth annotations.The loss compares ground-truth and predicted depth values.
  • B.2 Implementation Details: Surface-normal prediction uses inverse cosine similarity to align predicted unit normals with ground-truth unit vectors.The formulation uses the dot product between the two normal vectors.
  • B.2 Implementation Details: Edge detection uses class-balanced binary cross-entropy to address the extreme sparsity of edge pixels.The positive weight β is set to 0.95 to increase the penalty for missing edge pixels.

B.3 Metric for Each Task

Evaluation metrics vary by dataset and task, using overlap and accuracy measures for segmentation, error measures for depth, and angular measures for surface normals. The experiments also report ablations over smoothing and regularization parameters.

  • B.3 Metric for Each Task: NYUv2 evaluation reports mIoU and Pixel Accuracy for segmentation, Abs Err and Rel Err for depth, and angular errors and threshold percentages for surface normals.Surface-normal metrics include mean and median angular errors plus percentages within 11.25, 22.5, and 30 degrees.
  • B.3 Metric for Each Task: The experiments include a normalized baseline comparison and tables examining λ effects on convergence, Pareto-front quality, and convergence speed.The supplied table captions identify these evaluation scopes without reporting cell values.
  • B.3 Metric for Each Task: PASCAL-Context uses mIoU for semantic segmentation, human-parts segmentation, and saliency, and mean angular error for surface normals.The supplied passage identifies these task-specific metrics for the dataset.
  • B.3 Metric for Each Task: An ablation table evaluates the impact of τ on NYUv2, while reported settings use τ=5 for NYUv2 and τ=3 for Cityscapes and PASCAL-Context.The text states that performance is relatively insensitive to τ within a reasonable range.

C Additional Experimental Studies

Additional experiments show that SIMS benefits from principled scale-invariant scalarization rather than simple normalization, and generalizes beyond vision tasks.

  • D.1 Normalization baselines: 25.44% vs. 24.52%: EW with EMA improves over vanilla EW but still underperforms SIMS on NYUv2.STCH with epoch-wise normalization shows no gain, with gain=-0.02%.
  • D.1 Normalization baselines: Normalization alone does not guarantee a better trade-off and can disturb the original learning objective, whereas SIMS gains from principled scale-invariant scalarization with weak Pareto properties.
  • Non-vision experiment: 0.5884 average AUC: SIMS achieves the best result on Ali-CCP, mainly by improving the smaller-scale CVR task.The CVR loss is 1–2 orders of magnitude smaller than CTR, supporting evaluation beyond vision tasks.

D.3 Synthetic Experiments on 𝜆

Synthetic experiments examine how λ affects convergence and solution fidelity. Larger λ improves conditioning and stabilization speed, while smaller λ more faithfully approximates the original merit objective and often reaches the Pareto front more closely.

  • D.3 Synthetic Experiments on λ: Mean Rank=1.75: λ=0 achieves the smallest distance to the Pareto front in most tested benchmark and initialization cases.The experiments use BBOB problems F1, F2, and F11 with five λ values and four initializations.
  • D.3 Synthetic Experiments on λ: Larger λ generally stabilizes optimization faster, as the mean first hitting iteration for r_k<0.1 decreases with λ.The residual r_k measures proximity to a first-order stationary point.
  • D.3 Synthetic Experiments on λ: Choosing λ trades convergence rate against fidelity to the original merit objective, and the paper leaves principled or adaptive selection for future work.Positive λ improves inner-problem conditioning, whereas smaller λ often yields a smaller Pareto-front distance.
  • D.4 Rescale task losses on a real benchmark: Under 100× depth-loss rescaling on NYUv2, SIMS remains stable, with Δ_b changing from 25.44% to 25.21%.The comparison provides real-data evidence consistent with SIMS's scale-invariance claim.
Loading 2609.12599v1…