PaperlayerView exact revision on arXiv

Source-linked AI summary

Bounded Precision-Geometry Scaling for Robust Multi-Task Learning under Loss Scale Mismatch

Krishna Subedi

arXiv:2608.21653v1cs.LGcs.CV

TL;DR

Multi-task learning lacks robust weighting when task losses differ by orders of magnitude. BPGS uses bounded, batch-anchored uncertainty scaling with separated optimisation paths, and remains stable across extreme rescaling while performing strongly across several benchmarks.

  • Problem

    Large loss-scale disparities can make homoscedastic uncertainty weighting degrade severely during multi-task optimisation.

  • Method

    BPGS maps latent task uncertainties into a bounded chart anchored to detached batch log-loss statistics and separates network and uncertainty optimisation.

  • Results

    BPGS’s macro score changes by less than 0.01 across ×1–×1000 rescaling and delivers the best depth metrics and lowest total loss on NYUv2, with leading or competitive tabular performance.

  • Takeaways & Limitations

    BPGS provides a targeted scale-robust loss-weighting mechanism that complements existing gradient optimisers when loss-scale disparities dominate optimisation.

  • Takeaways & Limitations

    Evaluation is focused on NYUv2, two tabular benchmarks, and synthetic scaling tests, with primary benchmark results based on three random seeds.

Abstract

from arXiv · show

Multi-task learning often combines losses that span several orders of magnitude, causing homoscedastic uncertainty weighting to degrade severely. We propose Bounded Precision-Geometry Scaling (BPGS), a method that maps each task's log-variance through a bounded sigmoid parameterisation anchored to detached batch loss statistics, and decouples network optimisation from uncertainty optimisation. Its normalised task weights are provably invariant to uniform rescaling under non-degenerate loss scales. We evaluate BPGS on synthetic stress tests and three real-world benchmarks: NYUv2 dense prediction, Yeast multi-label classification, and RF1 multi-target regression. Under pure loss rescaling from $\times 1$ to $\times 1000$, its macro score changes from 0.777 to 0.778, whereas Kendall weighting drops from 0.780 to 0.637; $\ell_1$-normalising Kendall's weights does not close the gap. On NYUv2, BPGS records the lowest depth absolute relative error (0.223), depth RMSE (0.790), and total loss (1.891) among all compared methods, including Nash-MTL. Sensitivity studies on batch size and calibration show small variation across the tested ranges, and runtime overhead relative to Kendall is under 1%. BPGS posts the highest Yeast micro-F1 (0.616) and is competitive on RF1, though PCGrad leads RMSE and MAE there. These findings establish BPGS as a scale-robust alternative to homoscedastic uncertainty weighting, notably effective when loss-scale disparities dominate multi-task optimisation.

1 Introduction

Multi-task loss-scale disparities can let large-loss tasks dominate shared optimisation and destabilise uncertainty weighting. BPGS addresses this with bounded, batch-anchored uncertainty parameters and split optimisation, retaining stability under rescaling while improving selected benchmark outcomes.

  • Large raw-loss differences can dominate shared gradients, diminish smaller-loss task updates, and destabilise learned weighting parameters.
  • Homoscedastic uncertainty weighting is effective for similarly scaled losses but can produce destabilising precision shifts under severe disparities.PCGrad addresses directional gradient conflicts but does not solve scalar loss weighting.
  • BPGS bounds each task’s learned uncertainty using a sigmoid chart anchored to detached batch log-loss statistics and separates weight adaptation from network training.
  • Normalized BPGS task weights are strictly invariant to uniform loss rescaling under non-degenerate loss scales.
  • Across ×1–×1000 loss scaling, BPGS remains stable while standard homoscedastic weighting degrades markedly, and ℓ1 normalisation does not recover that stability.
  • On NYUv2, BPGS outperforms competitive baselines including Nash-MTL in depth accuracy and aggregate multi-task loss, with ablations examining its main design choices.
  • BPGS is additionally evaluated for batch-size, calibration, and runtime sensitivity, with gains extending to Yeast and RF1 tabular benchmarks.

2 Related Work

Related work spans adaptive loss weighting and gradient-space conflict resolution. The comparison set is scoped specifically to robustness against loss-scale disparities rather than comprehensive multi-task optimisation coverage.

  • Homoscedastic uncertainty weighting learns one log-variance per task, while GradNorm and Dynamic Task Prioritization adapt weights using training dynamics.
  • MGDA frames multi-task learning as multi-objective optimisation, while PCGrad and CAGrad modify gradient directions to reduce task conflict or negative transfer.
  • The comparison set covers both loss-weighting and gradient-manipulation paradigms, but is deliberately limited to the scale-robustness question.

3 Method

BPGS addresses loss-scale mismatch by anchoring bounded task log-variances to detached batch statistics and separating network and uncertainty updates. Its bounded precisions yield normalised weights that are invariant to uniform loss rescaling under non-degenerate conditions, while the guarantee remains batch-conditional.

  • 3.2 Canonical BPGS chart: BPGS learns one uncertainty coordinate per task and maps it into a bounded, batch-anchored log-variance chart.Detached batch log-loss statistics centre and scale the chart, restricting each task’s log-variance to a finite interval.
  • 3.3 Split optimisation objectives: BPGS separates network training from uncertainty adaptation using detached weights, losses, and batch statistics.The network pass updates shared parameters, while the uncertainty pass updates latent coordinates without coupling the two gradients within one step.
  • 3.2 Canonical BPGS chart: The radius of the bounded chart follows from the geometry of a population-standardised T-vector.The derivation bounds one coordinate by considering the minimum squared sum of the remaining T − 1 coordinates subject to zero mean.
  • 3.2 Canonical BPGS chart: The bounded chart produces positive precisions that are ℓ1-normalised into task weights.The finite interval for each log-variance implies bounded positive precision for every fixed batch.
  • 3.4 Batch-conditional boundedness: Boundary learnability is supported empirically but is not formally guaranteed for arbitrary tasks or training regimes.Logged runs avoided boundary locking, but sigmoid derivatives can vanish near the chart limits and recovery may stall.
  • 3.5 Uniform rescaling invariance: For uniformly rescaled losses L′_i = cL_i, normalised BPGS weights remain unchanged when loss and variance floors are non-degenerate.Rescaling adds log c to every batch log-loss, multiplying all raw precisions by 1/c; ℓ1 normalisation cancels that common factor, and first-batch calibration is unchanged.
  • 3.5 Uniform rescaling invariance: The invariance theorem concerns weights on a single batch, not complete optimiser trajectories or final predictive metrics.The broader effects of rescaling are evaluated empirically rather than established by the proposition.

4 Experimental Setup

The experiments span dense prediction, tabular classification and regression, synthetic scale stress, controlled ablations, and runtime sensitivity, using specified baselines, metrics, seeds, and final-epoch evaluation.

  • Benchmarks: NYUv2 evaluates semantic segmentation, depth estimation, and surface normals on 795 training and 654 validation images.The main comparison runs for 120 epochs and aggregates final-epoch metrics over seeds.
  • Controlled studies: NYUv2 ablations isolate bounded-chart and initialisation effects across four BPGS variants versus Kendall uncertainty weighting.The ablation uses a fixed 50% training subset, runs for 60 epochs, and uses three training seeds.
  • Benchmarks: Yeast is a 14-label classification benchmark, while RF1 is an 8-target regression benchmark for testing generalisability beyond dense prediction.Yeast has 1,500 training and 917 validation examples; RF1 has 4,108 training and 5,017 validation examples.
  • Stress tests: Synthetic studies test pure rescaling, scale stress, and heterogeneous interactions across multipliers ×1, ×10, ×100, and ×1000.The heterogeneous study adds clean, conflict, and noisy regimes, with macro score as the primary statistic and worst-task score for heterogeneous analysis.
  • Controlled studies: A diagnostic compares BPGS, Kendall, and ℓ1-normalised Kendall weights to assess whether normalisation alone explains rescaling behaviour.The diagnostic uses seeds 42, 123, and 999 and is reported separately from the main rescaling table.
  • Protocol: Evaluation records benchmark-specific task metrics, aggregates values across seeds, and uses final-epoch metrics for paper tables.Controlled studies also measure batch-size and calibration sensitivity, runtime, and peak CUDA memory; final NYUv2 checkpoints use val/miou, while ablations use val/total_loss.

5 Results

BPGS is robust to pure loss rescaling and severe scale stress, while retaining strong NYUv2 and Yeast performance with minimal overhead. Its advantage is narrower under conflict-dominated stress, and benchmark results do not establish universal method dominance.

  • Synthetic robustness: 0.777 to 0.778: BPGS’s macro score remains nearly unchanged under pure loss rescaling from ×1 to ×1000.Kendall falls monotonically to 0.637 at ×1000, while BPGS changes by only 0.001.
  • Synthetic robustness: 0.105: ℓ1-normalised Kendall still loses this much from ×1 to ×1000, whereas BPGS falls by 0.004.The diagnostic uses a different seed set from the main rescaling table, so it supports insufficiency of normalisation rather than a seed-matched gap estimate.
  • Synthetic robustness: 0.543 at ×1000: BPGS leads Kendall at 0.499 and UWSO at 0.494 under scale stress.All methods degrade as the scale-stress factor increases; the BPGS–Kendall gap is 0.044 at ×1000 and is not a formal significance test.
  • Scope and limitations: BPGS is less favourable in heterogeneous mixed-stress settings: Kendall and PCGrad score higher in noisy and conflict regimes.Its rescaling advantage is therefore limited to scale-dominated settings and does not extend to conflict-dominated ones.
  • NYUv2 benchmark: 0.223 absolute relative error, 0.790 depth RMSE, and 1.891 total loss: BPGS leads these NYUv2 metrics, while Static and UWSO lead other metrics.Relative to competing values, BPGS improves absolute relative error from 0.228 under UWSO, RMSE from 0.808 under Kendall, and total loss from 1.936.
  • Ablations: The ablation shows that the batch-aware chart and split objective materially affect stability and performance.The stateless auto-calibrated variant collapses, while removing the split objective increases uncertainty variability and lowers segmentation mIoU.
  • Transfer benchmarks: 0.616 micro-F1: BPGS is best on all three reported Yeast metrics, but PCGrad leads RF1 RMSE and MAE.On RF1, BPGS posts the second-best R2 at −0.429, with RMSE and MAE within about 2% of the best values.
  • Efficiency and sensitivity: 0.15% runtime and 0.87% peak-memory increases: BPGS adds little overhead relative to Kendall on the NYUv2 50% subset.The comparison reports 40.979 versus 40.916 seconds per epoch and 3368 versus 3339 MB peak GPU memory.

6 Discussion and Limitations

BPGS meets its scale-robustness objectives across controlled rescaling and standard benchmarks, while the study identifies important empirical and theoretical boundaries on its claims.

  • BPGS provides scale-robust loss weighting while complementing rather than replacing gradient optimisers.Controlled rescaling tests verify invariance under extreme loss-scale disparities, while standard benchmarks show no worsening of general task performance.
  • The evaluation covers NYUv2, two tabular benchmarks, and a synthetic scaling experiment, leaving broader dense-prediction validation for future work.
  • Three-seed primary benchmark results limit definitive fine-grained ranking claims among closely performing baselines.Scaling analyses use 10 seeds, but the primary benchmark results use three.
  • BPGS can face gradient saturation near chart boundaries, although experiments reached |zi|/τT = 0.93 without observed saturation.The theoretical analysis establishes batch-conditional boundedness and normalized-weight invariance, not formal convergence rates or asymptotic optimality.
  • Runtime measurements are restricted to a controlled NYUv2 subset and one GPU setting, while downstream deployments retain task-specific data, labeling, and evaluation risks.Aggregate scores can obscure improvement in one task alongside weakening in another, motivating per-task reporting.

7 Conclusion

BPGS is a bounded, batch-anchored uncertainty-weighting method for disparate loss scales, with invariant normalized weights and stable performance across extreme rescaling. Across benchmarks, it achieves strong aggregate and depth results while other methods lead on some tasks.

  • BPGS guarantees normalized task-weight invariance to uniform loss rescaling under non-degenerate loss scales.
  • Macro scores shift by less than 0.01 across ×1–×1000 rescaling, demonstrating greater stability than existing baselines under severe scale disparities.
  • BPGS achieves the lowest aggregate multi-task loss and best depth metrics on NYUv2, while other methods lead on segmentation and normals.
  • BPGS yields leading or highly competitive performance on Yeast and RF1 tabular tasks.
  • BPGS is designed for regimes where loss-scale disparity, rather than directional gradient interference, drives optimization instability and does not claim a universal ranking.Hybrid evaluation with conflict-aware optimizers remains future work.

A Reproducibility Details

The reproducibility setup specifies seeds, benchmark protocols, optimization safeguards, dataset configurations, baselines, and stress-test designs, with results reported using defined aggregation procedures.

  • Real-world benchmarks use three random seeds and final recorded epochs, while synthetic scale-stress uses 10 seeds and per-seed summaries.
  • BPGS detaches batch statistics and normalized weights during network updates, detaches task losses during uncertainty updates, and uses fixed gradient scale gθ = 100.Numerical floors are εlog = 10^-8 and εstd = 10^-4, with gradient clipping 10.0 for BPGS and 1.0 for Kendall.
  • NYUv2 uses 795 training and 654 validation images in a 3-task SegNet protocol, with a 50% subset for ablations.
  • Yeast is a 14-label classification benchmark and RF1 is an 8-target regression benchmark, both using shared-trunk MLPs.
  • Synthetic stress tests vary task count and correlation regimes, while baselines include a GradNorm-inspired gradient-norm proxy and analytical inverse-loss weighting.
  • Additional NYUv2 studies assess stop-gradient coupling, batch size, first-batch calibration, overhead, Nash-MTL, and normalization across controlled protocols.
  • The supplementary materials provide source code, configurations, and reproduction instructions.

C Ablation and NYUv2 Figures

The diagnostic studies clarify the behavior and scope of BPGS comparisons: Nash-MTL coefficients vary substantially, while batch and overhead tests remain controlled rather than universal.

  • Nash-MTL’s per-step coefficients varied substantially across batches in all three runs.The observation is descriptive and does not distinguish solver variation from an intrinsic benchmark property.
  • Batch-size and first-batch studies are controlled diagnostics on the NYUv2 subset, not claims about all dataset sizes, batch regimes, or initialization procedures.
  • Runtime measurements quantify the controlled BPGS–Kendall comparison on a single GPU setting without establishing a hardware-independent efficiency guarantee.

G Bounded-Chart Behavior

BPGS shows bounded, scale-robust behavior across synthetic stress tests and NYUv2, while retaining task-specific trade-offs and sensitivity to key design choices. Batch-aware calibration and stop-gradient support stronger trajectories, with small reported sensitivity and low overhead.

  • Stress behavior: L1-normalizing Kendall improves its largest-scale result but does not reproduce BPGS’s scale sensitivity.The normalization ablation therefore leaves a gap between Kendall and BPGS under pure loss rescaling.
  • Stress behavior: BPGS retains the strongest macro score under synthetic scale stress and is nearly invariant under pure loss rescaling.Its macro-score degradation is smallest under scale stress and essentially absent under pure rescaling.
  • Stress behavior: BPGS achieves the strongest worst-task score in the clean heterogeneous regime but is not best in every regime.Across regimes, it remains well above the degenerate UWSO worst-task score.
  • NYUv2 behavior: Batch-aware BPGS variants maintain stronger NYUv2 trajectories than the stateless auto-calibrated variant, whose segmentation and depth performance remain weak.Their task weights stay in distinct but controlled regimes, unlike the stateless fixed variant’s different long-run allocation.
  • Robustness and implementation: Reported sensitivity and implementation checks show limited variation across batch size and first-batch calibration, under-1% overhead relative to Kendall, and no run reaching the bounded-chart boundary.Logged sigmoid derivatives also remain empirically away from saturation, although this is not presented as a general guarantee.
  • NYUv2 behavior: On NYUv2, BPGS is strongest on both depth metrics, while Static leads segmentation and UWSO leads surface normals.The aggregate comparison also identifies BPGS as strongest among the adaptive methods against Static.
Loading 2608.21653v1…