PaperlayerView exact revision on arXiv

Source-linked AI summary

Gradient-Update Mismatch: Rethinking Conflict-Free Training of Physics-Informed Neural Networks

Jing Xiao, Xinhai Chen, Qinglin Wang, Menghan Jia, Zhiquan Lai, Dongsheng Li, Jie Liu, Tiejun Li

arXiv:2609.01558v1cs.LG

TL;DR

Gradient-surgery directions may lose conflict-freeness after optimizer transformation, so they do not guarantee conflict-free parameter updates. Gradient-Update Alignment projects optimizer proposals onto the conflict-free cone; across evaluated PINN settings, it achieves conflict-free applied updates and improves gradient-surgery methods.

  • Problem

    Gradient-surgery directions can leave the conflict-free cone after optimizer transformation, so pre-transformation conflict-freeness does not guarantee conflict-free parameter updates.

    Source: §1 · p. 2, §3.1 · p. 3

  • Method

    Gradient-Update Alignment projects the optimizer proposal onto the conflict-free cone and adjusts optimizer state toward targets reconstructed from the applied update.

    Source: §1 · p. 2, §3.4 · p. 5

  • Results

    Across evaluated PINN settings, GUA achieves conflict-free applied updates and consistently improves various gradient surgery methods.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Because the optimizer proposal determines the actual parameter-update direction, conflict-freeness must be considered at the update level, not only for the gradient-surgery direction.

    Source: §3.1 · p. 3

    The formulation uses local first-order gradient geometry, omitting curvature, finite-step, and longer-horizon effects; its hard conflict-free criterion may also exclude directions beneficial beyond that view.

    Source: §5 · p. 9

Abstract

from arXiv · show

Training Physics-Informed Neural Networks (PINNs) requires jointly optimizing physics residual and initial/boundary condition loss terms, which often induce conflicting gradients. Gradient surgery methods mitigate this issue by constructing directions from loss-specific gradients to reduce conflict before optimizer transformation. However, even when the constructed direction is conflict-free, this property may not be preserved after optimizer transformation. Let $a_t$ denote the direction constructed by gradient surgery, $u_t$ the optimizer proposal, and $\mathcal{C}_t$ the conflict-free cone induced by the loss-specific gradients. We show that modern optimizers can transform $a_t$ through mechanisms such as historical state, adaptive scaling, preconditioning, or decoupled weight decay, so $a_t \in \mathcal{C}_t$ does not generally imply $u_t \in \mathcal{C}_t$. We refer to this optimizer-induced discrepancy in conflict-freeness between $a_t$ and $u_t$ as Gradient-Update Mismatch (GUM). Accordingly, we propose Gradient-Update Alignment (GUA), which projects $u_t$ onto $\mathcal{C}_t$ to obtain the aligned update $p_t$ and applies $p_t$ to the parameters. When the optimizer maintains internal state, GUA further adjusts this state toward targets reconstructed from the applied update. We conduct extensive experiments and find that GUM is widespread across momentum, adaptive, and curvature-based optimizers, with conflict rates reaching up to 86.3%. Across all PINN settings, GUA achieves conflict-free applied updates and consistently improves various gradient surgery methods, reducing the relative $L_2$ error by up to 98.2% in individual settings. Data and code are available at https://github.com/JingXiao10/GUA.

1 INTRODUCTION

PINN loss gradients can conflict, and a direction made conflict-free by gradient surgery may become conflicting after optimizer transformation. The introduction presents GUA, which aligns the optimizer proposal with the conflict-free cone before applying the update.

1 INTRODUCTION

  • PINN gradients from different loss terms can differ in magnitude and direction, while loss balancing addresses magnitude imbalance but not directional conflict.Gradient surgery modifies loss-specific gradients to construct directions with reduced conflict.Source: §1 · p. 1
  • Gradient–Update Mismatch (GUM) occurs when optimizer transformation moves a gradient-surgery direction outside the loss-gradient-induced conflict-free cone.The introduction identifies historical state, adaptive scaling, preconditioning, and decoupled weight decay as possible causes.Source: §1 · p. 2
  • GUA projects the optimizer proposal onto the current conflict-free cone and applies the resulting aligned update to the parameters.When the optimizer maintains internal state, GUA adjusts it toward targets reconstructed from the applied update to reduce the influence of components removed by projection on subsequent proposals.Source: §1 · p. 2
  • The paper derives exact conflict-preservation conditions for fixed-state affine transformations and extends its analysis to general nonlinear optimizer maps.The analysis covers historical state, adaptive scaling, preconditioning, and weight decay.Source: §1 · p. 2
  • GUA achieves conflict-free applied updates across evaluated PINN settings and consistently improves various gradient surgery methods.The experiments cover momentum, adaptive, and curvature-based optimizers.Source: §1 · p. 2

2 RELATED WORK

PINN training challenges include imbalanced gradients, poor conditioning, and conflicting loss directions; existing gradient surgery acts before optimizer transformation. The related work motivates extending conflict handling to the optimizer proposal before parameter application.

2.1 PINN TRAINING AND LOSS BALANCING

  • Loss balancing addresses imbalanced PINN training only partly because reweighting does not directly resolve conflicts between loss-specific gradient directions.PINNs can also be difficult to train because of poor conditioning and differing convergence rates across loss terms (Krishnapriyan et al., 2021).Source: §2.1 · p. 2

2.2 GRADIENT SURGERY IN MULTI-LOSS OPTIMIZATION

2.3 OPTIMIZER TRANSFORMATIONS AND UPDATE-LEVEL ALIGNMENT

  • Optimizer transformations can alter surgery-constructed directions through historical state, adaptive scaling, preconditioning, or decoupled weight decay, potentially disrupting conflict-free geometry.The goal is not a new optimizer, but extending conflict handling to the optimizer proposal before it is applied to parameters; examples include momentum SGD, RMSProp, Adam, AdamW, Sophia/SophiaG, AdaHessian, and SOAP.Source: §2.3 · p. 3

3 GRADIENT–UPDATE MISMATCH AND GRADIENT–UPDATE ALIGNMENT

A conflict-free direction from gradient surgery may become conflicting after optimizer transformation, motivating GUA: project the optimizer proposal onto the conflict-free cone and, when projection changes it, align optimizer state toward targets based on the applied update.

3.1 PRELIMINARIES: MULTI-LOSS OPTIMIZATION

  • The conflict-free cone C_t contains directions whose inner product with every loss-specific gradient is nonnegative, preventing first-order loss increases.C_t characterizes the current conflict-free geometry used for update-level diagnosis and projection.Source: §3.1 · p. 3

3.2 GRADIENT–UPDATE MISMATCH

  • GUM occurs when a conflict-free constructed direction a_t produces an optimizer proposal u_t outside the conflict-free cone C_t.When gradient surgery does not guarantee conflict-free directions, the rate-level diagnostic is R_u > R_a, indicating a net increase in conflicting proposals.Source: §3.2 · p. 4

3.3 CONFLICT PRESERVATION UNDER OPTIMIZER TRANSFORMATIONS

  • Affine optimizer transformations need not preserve conflict-freeness: positive coordinate-wise scaling can produce u_t outside C_t even when a_t is inside.The example gives g_2,t · u_t = −4.9; historical state, adaptive scaling, preconditioning, and weight decay can also cause this mismatch.Source: §3.3 · p. 4

3.4 GRADIENT–UPDATE ALIGNMENT

  • GUA projects u_t onto C_t to obtain p_t, which is unchanged if u_t is already conflict-free and is the closest conflict-free update under the chosen metric.For locally smooth losses and a conflicting proposal, the projected update's maximum positive loss change is O(η_t^2), while the proposal has a positive first-order violation for sufficiently small η_t.Source: §3.4 · p. 5
  • When projection modifies the proposal, GUA softly aligns optimizer state toward targets reconstructed from p_t; otherwise, it retains the tentative state.For Adam, the target state is reconstructed from a pseudo-gradient consistent with the applied update under the current adaptive denominator; the approach can extend to other stateful optimizers.Source: §3.4 · p. 5

4 EXPERIMENTS

Across six PDE benchmarks and a CelebA task-cardinality benchmark, experiments test whether optimizer transformations reintroduce conflicts after gradient surgery and whether GUA improves accuracy, alignment, and efficiency. Results also examine GUA’s state-alignment mechanism and performance as task counts increase.

4.1 EXPERIMENTAL SETUP

  • Across six PDE benchmarks, GUA is evaluated with six gradient-surgery methods in two- and three-loss settings, alongside a CelebA benchmark varying task counts.PINN results average five runs on one RTX 4090 GPU; CelebA results average three runs. The six methods are PCGrad, CAGrad, IMTL-G, A-MTL, UPGrad, and ConFIG.Source: §4.1 · p. 6

4.2 Q1: OPTIMIZER-INDUCED GUM

  • Plain SGD preserves Ru = 0, but M-SGD yields Ru = 51.3%–86.3%, showing that optimizer transformations can reintroduce conflicts after ConFIG.ConFIG has Ra = 0 in all settings, so nonzero post-optimizer Ru indicates GUM; adaptive and curvature-based optimizers also produce nonzero Ru at rates that vary across PDEs.Source: Table 1 · p. 6, §4.2 · p. 6

4.3 Q2: PINN PERFORMANCE

  • Rp = 0 across all PINN settings, and GUA improves accuracy across gradient-surgery methods and PDE benchmarks.Relative L2 error reductions range from 11.6% to 98.2%, with median reductions of 42.2% and 61.9% in the two- and three-loss settings, respectively. Adam+GUA improves over Adam in all 10 settings and outperforms standalone gradient-surgery baselines in four.Source: §4.3 · p. 7, Figure 3 · p. 7

4.4 Q3: DIRECTION ALIGNMENT MECHANISM

  • In 95.1% of paired comparisons GUA’s update has lower worst relative one-step loss change than the optimizer proposal, and in 93.4% than the norm-matched control.The norm-matched control preserves the optimizer proposal’s direction while matching the GUA update’s norm, isolating the effect of direction alignment.Source: §4.4 · p. 7, Figure 4 · p. 8

4.5 Q4: STATE ALIGNMENT MECHANISM

  • Soft state alignment further improves on direction alignment alone in most settings, while exact alignment is less effective and exact alignment of both Adam moments can degrade performance or become unstable.The authors suggest that exact replacement can discard useful history and alter future adaptive scaling; soft alignment preserves some history while moving state toward update-reconstructed targets.Source: §4.5 · p. 8

4.7 Q6: TASK-CARDINALITY SCALING

  • As CelebA task cardinality increases from 2 to 40, GUA improves F1 in six of seven settings and achieves comparable performance at 20 tasks.Additional memory cost reaches only 4.2% at 40 tasks, while training-time overhead becomes more pronounced for larger task counts.Source: Figure 7 · p. 9, §4.7 · p. 9

4.6 Q5: GUA OVERHEAD

  • GUA consistently improves accuracy across PINN settings at the cost of additional training time, while adding little peak GPU memory overhead.The projection uses an m-dimensional dual rather than a p × p matrix, with O(m^2p) Gram-matrix construction; for m = 2 or 3, active-set enumeration solves the dual exactly.Source: §4.6 · pp. 8–9, Figure 6 · p. 9

5 CONCLUSION … A.2 GENERAL NONLINEAR CONFLICT PRESERVATION

The paper concludes that GUA enforces conflict-free updates and improves PINN performance, while its current first-order, hard-constraint formulation leaves curvature and longer-horizon effects for future work. The appendix characterizes when affine and nonlinear optimizer transformations preserve conflict-freeness.

5 CONCLUSION

  • GUA enforces conflict-free geometry at the update level, eliminates update-level conflicts, and substantially improves PINN performance across evaluated settings.The work identifies GUM through theoretical analysis and experiments, then proposes GUA to address it.Source: §5 · p. 9
  • The current GUM/GUA formulation captures local first-order gradient geometry, not curvature, finite-step, or longer-horizon effects.The authors identify extensions beyond instantaneous first-order geometry and softer conflict criteria as future work, since the hard criterion may exclude directions beneficial beyond the local view.Source: §5 · p. 9

A THEORY AND PROOFS

  • For affine optimizer maps, conflict preservation is characterized by their linear component mapping each loss gradient into the cone generated by the loss gradients.The proof’s sufficiency argument uses that the bias lies in the cone and that the convex cone is closed under addition.Source: §A.1 · p. 12

A.1 PROOF OF PROPOSITION 1

  • The affine proof establishes sufficiency by showing that the transformed direction lies in the cone when each transformed loss gradient and the bias lie in that cone.The cone’s closure under addition yields the conflict-preserving affine map.Source: §A.1 · p. 12

A.2 GENERAL NONLINEAR CONFLICT PRESERVATION

  • For locally Lipschitz nonlinear optimizer maps, conflict preservation is characterized by a ray-wise derivative condition for every cone direction and loss gradient, holding almost everywhere.The characterization examines the map along ϕ_t,a(τ) = T_t(τa), for τ in [0, 1].Source: §A.2 · p. 13
  • The locally Lipschitz characterization covers piecewise-smooth optimizer maps because ray-wise derivatives need only exist almost everywhere, including with clipping-induced nondifferentiabilities.Source: §A.2 · p. 13
  • For continuously differentiable maps, the nonlinear condition is expressed using the optimizer map’s Jacobian, and the affine case recovers Proposition 1.In the affine case, the map’s Jacobian is constant and its value at zero is the bias.Source: §A.2 · p. 13

A.3 PROOF OF PROPOSITION 2 · A.4 NORM MATCHING DOES NOT RESTORE CONFLICT-FREENESS · B EXPERIMENTAL DETAILS FOR PINNS

The proof shows that projecting a conflicting optimizer proposal into the conflict-free cone yields a least-change feasible update with a strict small-step improvement guarantee under stated smoothness and neighborhood conditions. The norm-matching observation shows that rescaling alone cannot restore conflict-freeness; changing direction is necessary.

A.3 PROOF OF PROPOSITION 2

  • The projected update lies in the conflict-free cone, so it has no positive first-order loss-change term for any loss.Cone membership entails nonnegative inner products with the loss-specific gradients.Source: §A.3 · p. 14
  • For sufficiently small admissible steps, the projected update strictly improves on a conflicting optimizer proposal under the stated assumptions.The losses must be smooth on a convex neighborhood, and both trial points must remain in that neighborhood.Source: §A.3 · pp. 14–15
  • GUA makes the smallest feasible correction under the M_t metric, and the projected update is closer than the optimizer proposal to every feasible direction, including a_t when a_t is feasible.Source: §A.3 · pp. 15–16
  • If the projected update is strictly inside the cone, sufficiently small steps decrease all losses simultaneously; on the boundary, only the absence of a positive first-order loss-change term is guaranteed.Source: §A.3 · p. 16

A.4 NORM MATCHING DOES NOT RESTORE CONFLICT-FREENESS

  • Positive rescaling cannot restore conflict-freeness, and the norm-matched control remains conflicting despite sharing the projected update’s Euclidean norm.Source: §A.4 · pp. 16–17
  • Norm matching alone cannot remove GUM; restoring conflict-freeness requires changing the update direction.Source: §A.4 · p. 17

B.1 PDE BENCHMARKS · B.2 NETWORK ARCHITECTURES

The PDE benchmarks span varied equation types, dimensions, and flow problems, while all models use shared-trunk MLPs whose architecture is held fixed across baseline and +GUA comparisons.

B.1 PDE BENCHMARKS

  • Benchmarks span steady and time-dependent PDEs, scalar and multi-component systems, low- and high-dimensional domains, and incompressible flows.Protocols are mainly based on PINNacle and ConFIG.Source: §B.1 · p. 18
  • The suite includes nonlinear Schrödinger, one-dimensional viscous Burgers, and steady two-dimensional Kovasznay-flow problems.Schrödinger uses a complex solution representation and periodic boundaries at x = −5 and x = 5.Source: §B.1 · p. 18
  • Other cases include multiscale two-dimensional Heat-MS data, unsteady three-dimensional Beltrami flow, and a scalar Poisson equation on a five-dimensional hypercube.Beltrami uses Re = 1 with a = d = 1; Poisson-5D is from PINNacle.Source: §B.1 · pp. 18–19

B.2 NETWORK ARCHITECTURES

  • Each benchmark uses a fully connected MLP that maps physical coordinates through a shared hidden representation to PDE-specific output channels.The hidden width is shared within a benchmark, and one trunk serves all loss terms, coupling residual, boundary, and initial-condition losses through the same parameters.Source: §B.2 · p. 20
  • Inputs follow each benchmark’s physical coordinates, outputs match its PDE state, and automatic differentiation computes the residual derivatives through the MLP.Outputs include multiple fields for complex or multi-species systems and velocity components with pressure when pressure appears in the residual.Source: §B.2 · p. 20
  • Baseline gradient-surgery methods and their +GUA variants use identical architectures, Xavier initialization, and matched initialization for each seed.Depth, width, activation, and output parameterization are not retuned for GUA, isolating its effect from architectural differences.Source: §B.2 · p. 20

B.3 TRAINING SETTINGS … C ADDITIONAL RESULTS, ABLATIONS, AND DIAGNOSTICS

The appendices specify PINN training and optimizer-baseline settings, evaluation metrics, and GUA implementation. GUA projects optimizer proposals in update space and can align optimizer state to the applied update.

B.3 TRAINING SETTINGS

  • PINN runs use Xavier initialization, full-batch training, equal loss weights, five seeds, and a 100-epoch linear warmup followed by cosine decay by default.Adam is the default optimizer; validation occurs every 100 epochs, and the best-validation checkpoint is tested. Heat-MS with three losses instead uses a constant learning rate of 10^-4 after warmup to avoid instability in baseline ConFIG.Source: §B.3 · p. 21

B.4 OPTIMIZER CONFIGURATIONS

  • The optimizer-induced GUM experiments fix ConFIG as the gradient-surgery method and apply each optimizer proposal directly, without GUA.ConFIG constructs the pre-optimizer direction; the section specifies how each optimizer transforms it into a proposal.Source: §B.4 · p. 22

B.5 BASELINE SETTINGS

B.6 EVALUATION METRICS

  • PINN benchmarks report relative L2 error and conflict rates across raw gradients, constructed directions, optimizer proposals, and aligned updates; baseline runs omit the aligned-update rate.The diagnostic sequence is Rg → Ra → Ru → Rp.Source: §B.6 · p. 24

B.7 GUA IMPLEMENTATION DETAILS

  • GUA projects the optimizer proposal onto the current conflict-free cone, applies the aligned update, and aligns optimizer state when projection changes the proposal.The implementation details correspond to Algorithm 1.Source: §B.6 · p. 25, §B.7 · p. 25

B.7.1 PROJECTION SOLVER

  • The projection operates on the flattened full-network update in the global cone, using the Adam metric by default; identity and diagonal Adam metrics are the experimentally used choices.The appendix describes the Euclidean metric as the optimizer-agnostic option and the diagonal Adam metric as optimizer-guided; layer-wise projection would impose different constraints.Source: §B.7.1 · p. 25
  • For PINN experiments with two or three losses, the projection is solved exactly by enumerating active sets in the low-dimensional dual.For more than eight losses, the appendix instead specifies an active-set solver for the dual quadratic program; the parameter dimension enters through gradient inner products.Source: §B.7.1 · p. 26

B.7.2 OPTIMIZER-STATE ALIGNMENT

  • GUA softly aligns Adam’s stored first- and second-moment states toward targets reconstructed from the applied update, and extends to stateful optimizers when their target states are reconstructible.The Adam instantiation derives a pseudo-gradient that would produce the applied update under the current denominator, then blends the tentative moments with target moments.Source: §B.7.2 · p. 27

C.1 PROJECTION METRIC ABLATION · C.2 STATE-ALIGNMENT SENSITIVITY

The projection-metric ablation finds the Adam metric at least competitive in accuracy and more numerically robust in difficult settings, while state-alignment performance is broadly robust across most evaluated equations, methods, and coefficient pairs.

C.1 PROJECTION METRIC ABLATION

  • Both projection metrics enforce the same conflict-free constraints but select different feasible updates; Adam’s metric accounts for coordinate-wise scaling through second-moment statistics.The Euclidean metric treats coordinates uniformly, whereas the Adam metric preserves optimizer-induced geometry; the paper uses Adam’s metric by default.Source: §C.1 · p. 28
  • The Adam metric is comparable to or more accurate than the Euclidean metric, and stays stable on Heat-MS where Euclidean projection is partly unstable or non-finite.The Euclidean metric is partly unstable in the two-loss Heat-MS setting and entirely non-finite in the three-loss setting; Adam remains stable in both.Source: Figure 8 · p. 28, §C.1 · p. 28

C.2 STATE-ALIGNMENT SENSITIVITY

  • The sensitivity analysis evaluates seven predefined coefficient pairs across all PINN settings, six gradient surgery methods, and five seeds.The coefficients ρm and ρv control how strongly Adam’s moments move toward the target state.Source: §C.2 · p. 29
  • Alignment preferences vary by equation: Schrödinger and Burgers favor moderate alignment, Heat-MS stronger alignment, while Kovasznay flow and Poisson-5D vary more by method.Source: §C.2 · p. 29
  • Coefficients are selected using validation performance and shared across loss decompositions and gradient surgery methods for each equation, without using the test set.Figure 9 retains the single numerical failure and highlights validation-selected coefficients.Source: §C.2 · p. 29, Figure 9 · p. 29
  • GUA’s relative-error ratios are below one across most equations, methods, and coefficient pairs, with Burgers, Beltrami flow, and Poisson-5D particularly robust.Figure 9 reports geometric means of paired relative-error ratios to the corresponding baseline; values below one indicate improvement.Source: Figure 9 · p. 29, §C.2 · p. 29

C.3 OPTIMIZER-INDUCED GUM RESULTS ON KOVASZNAY, BELTRAMI, AND POISSON-5D · C.4 FULL PINN RESULTS

Table 7 extends optimizer-induced GUM results to Kovasznay, Beltrami, and Poisson-5D, while C.4 reports complete Q2 results across evaluated method–benchmark configurations. The results distinguish conflict-free gradient-surgery directions from post-optimizer behavior and detail error and conflict-rate outcomes across training stages.

C.3 OPTIMIZER-INDUCED GUM RESULTS ON KOVASZNAY, BELTRAMI, AND POISSON-5D

  • Across Kovasznay, Beltrami, and Poisson-5D, momentum, adaptive, and curvature-based optimizers can produce nonzero post-optimizer conflict rates, while ConFIG has Ra = 0.Table 7 reports the same experimental protocol as the main-text subset; the sequence Rg → Ra → Ru tracks conflict rates from raw gradients through surgery to optimizer transformation.Source: §C.3 · p. 30, Table 7 · p. 30
  • Plain SGD preserves Ru = 0, as expected because its optimizer proposal ut equals the surgery direction at.Source: §C.3 · p. 30

C.4 FULL PINN RESULTS

  • C.4 provides complete Q2 results for each evaluated method–benchmark configuration, including relative L2 errors and conflict-rate transitions over four training stages.Source: §C.4 · p. 31
  • The complete Q2 summary covers 36 two-loss and 24 three-loss configurations for the six gradient-surgery methods, excluding the Adam control.Source: §C.4 · p. 31
  • Main-text Q2 summaries use configuration-level ranges and medians, with the reported medians taken over configuration-level reductions.Source: §C.4 · p. 31

C.4.1 QUANTITATIVE PINN RESULTS FOR Q2 … C.6.2 REPRESENTATIVE GRADIENT-DIRECTION SLICES

Across PINN settings, GUA reduces relative L2 error and produces conflict-free applied updates across different gradient-surgery methods. It also improves results with RMSProp, AdamW, and SOAP, while the diagnostics illustrate how optimizer transformations can disrupt conflict-free directions.

C.4.1 QUANTITATIVE PINN RESULTS FOR Q2

  • Across all evaluated base methods and PINN settings, adding GUA reduces mean error; Adam+GUA improves over Adam in all ten settings.Table 8 reports relative L2 errors as mean ± sample standard deviation over five seeds.Source: Table 8 · p. 31, §C.4.1 · p. 31

C.4.2 CONFLICT-RATE PINN RESULTS FOR Q2

  • GUA yields a 0% applied-update conflict rate in every evaluated Q2 setting, despite different conflict responses across gradient-surgery methods.With Adam fixed, the experiments vary the surgery method; when pre-optimizer conflict is zero, any nonzero proposal conflict indicates strict GUM, while an increase indicates generalized GUM.Source: Table 9 · p. 32, §C.4.2 · p. 32

C.5 PERFORMANCE ACROSS OPTIMIZERS

  • GUA projects proposals using optimizer-specific metrics and softly aligns optimizer state toward targets reconstructed from the applied update.The evaluation compares ConFIG+optimizer with ConFIG+optimizer+GUA for RMSProp, AdamW, and SOAP; SOAP’s preconditioning is kept unchanged during state alignment, which is applied only when GUA modifies the proposal.Source: §C.5 · p. 33
  • GUA reduces relative L2 error for RMSProp, AdamW, and SOAP across both Burgers loss decompositions.For Burgers, the reductions were 52.35%, 65.19%, and 13.78% for RMSProp, AdamW, and SOAP in the two-loss setting, respectively. In the three-loss setting, they were 58.39%, 62.92%, and 42.73%, respectively.Source: Figure 10 · p. 33, §C.5 · p. 33
  • GUA also reduces relative L2 error for RMSProp, AdamW, and SOAP across both Schrödinger loss decompositions.For Schrödinger, the reductions were 22.98%, 40.00%, and 1.70% for RMSProp, AdamW, and SOAP in the two-loss setting, respectively. In the three-loss setting, they were 47.50%, 43.66%, and 4.77%, respectively.Source: Figure 10 · p. 33, §C.5 · p. 33

C.6.1 SAME-STATE NORM-MATCHED COMPARISON

  • The same-state, norm-matched comparison holds parameters, optimizer state, batch, and learning rate fixed while comparing the proposal, aligned update, and norm-matched control.Table 10 reports per-benchmark Q3 win rates under two- and three-loss decompositions; all reported samples satisfy pt ≠ 0.Source: §C.6.1 · p. 34, Table 10 · p. 34

C.6.2 REPRESENTATIVE GRADIENT-DIRECTION SLICES

  • Representative two-loss slices show optimizer proposals moving outside the conflict-free cone and GUA realigning them within it.Figure 11 projects vectors into the span of the two loss gradients and normalizes them, so it shows relative directions and conflict-free geometry rather than original magnitudes.Source: §C.6.2 · p. 34, Figure 11 · p. 35

D MULTI-TASK LEARNING DETAILS

CelebA is used to study multi-task learning across nested task-count settings, with GUA applied to shared parameters after task-gradient aggregation and the optimizer proposal. The data pipeline, model, and task losses remain unchanged.

D MULTI-TASK LEARNING DETAILS

  • CelebA task-count settings use the first m attributes in official annotation order, forming nested subsets, with 64 × 64 images and no additional augmentation.The model uses a shared convolutional backbone and task-specific binary classification heads.Source: D MULTI-TASK LEARNING DETAILS · p. 36
  • GUA modifies shared parameters after task-gradient aggregation and the optimizer proposal, while task-specific classification heads use the native optimizer without GUA.The data pipeline, model, and task losses remain unchanged.Source: D MULTI-TASK LEARNING DETAILS · p. 36
  • Training minimizes binary cross-entropy independently for each attribute, and reported results average three seeds: {0, 1, 2}.The main training hyperparameters are summarized in Table 11.Source: D MULTI-TASK LEARNING DETAILS · p. 36

Related papers

Loading 2609.01558v1…