PaperlayerView exact revision on arXiv

Source-linked AI summary

Geometry-Aware Multi-Objective Optimization: Norm-Constrained Oracles with Entropic Weight Updates

El Mahdi Chayti, Alireza Mirrokni, Martin Jaggi

arXiv:2610.13773v1math.OCcs.LG

TL;DR

Multiple-gradient descent is typically Euclidean, whereas matrix-aware optimizers use directions induced by other norms. The paper analyzes a single-loop multi-objective method that combines a norm-constrained linear minimization oracle with entropic task-weight updates. Under a trajectory gradient bound, stochastic convergence takes O(ϵ−4) iterations; explicit K dependence is absent from the leading T−1/4 term.

  • Problem

    Multiple-gradient descent is typically Euclidean, whereas matrix-aware optimizers use directions induced by other norms.

    Source: §1 · p. 1

  • Method

    The paper analyzes a single-loop multi-objective method that combines a norm-constrained linear minimization oracle with entropic task-weight updates.

    Source: §1 · p. 2

  • Results

    Under a trajectory gradient bound, stochastic convergence takes O(ϵ−4) iterations; explicit K dependence is absent from the leading T−1/4 term.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A selection-independent inner gap addresses nonunique optimal blended gradients, and regularized tracking controls the inner gap at every iteration after burn-in.

    Source: §1 · p. 2

    The gradient bound is derived only for the deterministic unregularized case, tracking exponents are not competitive with Euclidean CA-distance results, and experiments use K ≤ 3 with τ = κ = 0.

    Source: §7 · p. 9

Abstract

from arXiv · show

Multiple-gradient descent is typically formulated in Euclidean geometry, whereas matrix-aware optimizers use directions induced by other norms. We analyze a single-loop multi-objective method combining a norm-constrained linear minimization oracle with entropic updates of task weights. For any finite-dimensional norm pair, we establish an $O(T^{-1/2})$ ergodic stationarity rate in the deterministic setting, deriving the required trajectory gradient bound from smoothness and lower-boundedness in the unregularized case. With stochastic gradients and a bounded-gradient assumption, one persistent blended-gradient momentum buffer yields an $O(T^{-1/4})$ rate whose leading term has no explicit dependence on the number of tasks. We also establish static regret for the inner weighting objective on arbitrary bounded gradient sequences and, with entropy regularization and a weight floor, iteration-wise inner-gap control after burn-in while retaining ergodic stationarity, conditional on a trajectory gradient bound. The optimal blended gradient can be nonunique in spectral and sign geometries, motivating a selection-independent inner gap. Our analysis extends the spectral construction studied by concurrent MOON to general norms. Experiments compare the method with existing multi-task optimizers and examine its oracle, weighting, and momentum choices.

1 INTRODUCTION

Multi-objective learning seeks Pareto stationarity, but Euclidean multiple-gradient descent does not reflect the non-Euclidean directions used by matrix-aware optimizers. This work extends entropic, norm-aware multi-objective optimization with deterministic and stochastic guarantees and inner-weight tracking results.

1 INTRODUCTION

  • Multiple-gradient descent targets Pareto stationarity by choosing the convex combination of task gradients with smallest norm.The motivating settings include shared-backbone tasks and language models balancing helpfulness against safety.Source: §1 · p. 1
  • The method extends the spectral construction of MOON to arbitrary norm pairs and adds a derived deterministic gradient bound, one-buffer stochastic analysis, and inner-problem tracking results.MSMD uses Euclidean directions and a double loop; MOON uses spectral-norm geometry and logit-based weighting.Source: §1 · p. 2
  • For the unregularized, unclipped method, smoothness and lower-boundedness yield a deterministic gradient bound and O(L∆log^2 K/ϵ^2) iteration complexity without a separate gradient-bound assumption.Each iteration uses K task-gradient evaluations, one oracle call, and one entropic weight update.Source: §1 · p. 2
  • With one persistent blended-gradient momentum buffer, stochastic stationarity takes O(ϵ^-4) iterations, with no explicit task-count dependence in the leading rate term.Under bounded variance, lower-order terms retain linear K dependence; a stronger tail assumption reduces it to log K.Source: §1 · p. 2
  • With regularization, clipping, and a trajectory gradient bound, the method controls the inner gap at every iteration after burn-in while retaining ergodic stationarity.The stated burn-in is approximately O(M^4 log^2 K/ϵ^4), with a non-competitive exponent.Source: §1 · p. 2
  • The method also gives static regret for the inner weighting objective on arbitrary bounded gradient sequences.The optimal blended gradient can be nonunique in Muon and Signum geometries, motivating a selection-independent inner gap.Source: §1 · p. 2

2 RELATED WORK

Prior work develops non-Euclidean optimizers mainly for single-objective problems, while stochastic multi-gradient analyses and the closest multi-objective methods retain Euclidean or bounded-gradient assumptions. The paper positions its contribution as generalizing norm-aware multi-objective analysis and refining inner-weight guarantees.

2 RELATED WORK

  • Geometry-aware optimizers use non-Euclidean steepest descent and linear minimization oracles, but the cited methods are single-objective.Examples include Muon, Scion, and the LMO formulation of Pethick et al. (2025).Source: §2 · p. 2
  • Prior stochastic MGDA analyses use Euclidean geometry with projected weights, with reported iteration-wise CA-distance rates of O(ϵ^-11) and O(ϵ^-17) in Zhang et al. (2025).That work reports O(ϵ^-2) and O(ϵ^-4) for average CA distance.Source: §2 · p. 2
  • MSMD combines entropic weights with a Euclidean direction and double loop, whereas MOON combines spectral–nuclear geometry with momentum and logit-based weighting.Both prior analyses assume bounded gradients; MOON also assumes bounded losses and has polynomial dependence on the number of objectives.Source: §2 · p. 2
  • The related oracle table concerns norm-constrained oracles, which Pethick et al. (2025) apply layer-wise with per-layer radii ρℓ.The supplied caption does not state the table's individual entries.Source: Table 1 · p. 3
  • The analysis uses established entropic mirror-descent bounds from Nemirovsky & Yudin (1983), Beck & Teboulle (2003), Kivinen & Warmuth (1997), and Bubeck (2015).Chen et al. (2026) is cited as a survey.Source: §2 · p. 3

3 SETUP

The setup defines multi-objective stationarity and inner optimality for an arbitrary primal–dual norm pair, then specifies how oracle choices and assumptions enter the analysis. It also distinguishes selection-independent inner-gap guarantees from claims about descent on every task.

3 SETUP

  • A primal norm measures parameter displacements, its dual measures gradients, and the LMO selects a feasible direction from a norm-bounded set.For block-structured parameters, the paper gives a layer-wise norm, dual, and oracle construction; the update adds the oracle direction to the current iterate.Source: §3 · p. 3
  • Pareto stationarity requires the minimum dual norm of a convex blend of task gradients to be small; the inner gap measures the used blend's excess norm over that minimum.The gap is zero exactly when the current weights solve the inner problem, and it does not depend on a choice among minimizers.Source: §3 · p. 3
  • If the dual unit ball is strictly convex, every minimizing weight gives the same optimal blended gradient; without strict convexity, that gradient can vary across minimizers.Strict convexity holds for ℓp with 1 < p < ∞, but fails for ℓ1 and the nuclear norm, including the Muon and Signum geometries discussed here.Source: §3 · p. 3
  • A stationary point of a fixed scalarization is Pareto stationary because its scalarized gradient is a zero blend of task gradients.The proposition states this for any fixed weighting u in the simplex.Source: §3 · p. 3
  • The actual blended-gradient residual upper-bounds average inner gap and CA distance, but this also holds for a fixed weighting and therefore does not establish tracking.The inner gap does not certify simultaneous per-task descent for every oracle selection; separate results provide comparator-dependent regret and post-burn-in iterate-wise tracking.Source: §3 · p. 4
  • The analysis assumes norm-pair smoothness and a stochastic oracle with conditional unbiasedness and bounded variance; a trajectory gradient bound is assumed only for the stochastic setting.In the deterministic analysis, the gradient bound is derived rather than assumed.Source: §3 · p. 4

4 ENTROPIC LMO-MGDA

Entropic LMO-MGDA combines a norm-constrained oracle with entropic task-weight updates in a single loop. Its analyses establish deterministic and stochastic ergodic stationarity guarantees, alongside inner-weighting guarantees under their stated conditions.

4 ENTROPIC LMO-MGDA

  • The entropic update descends on oracle scores, optionally pulls weights toward uniformity and enforces a floor, without requiring an inner optimization loop.At zero regularization and floor it is exponentiated gradient; a positive floor uses a clipping recurrence.Source: §4 · pp. 4–5

4.1 DETERMINISTIC ANALYSIS

  • In any finite-dimensional norm pair, deterministic stationarity reaches O(LΔ log²K/ϵ²) iterations when the unregularized trajectory bound is derived from smoothness and lower-boundedness.The rate gives an ϵ-accurate Pareto-stationary iterate.Source: §4.1 · p. 5
  • On arbitrary bounded gradient sequences, the weighting update has static regret, which decomposes into an average inner gap and comparator-variation term.Along the deterministic trajectory, stationarity already controls average gap; the regret decomposition is informative when comparator variation is controlled.Source: §4.1 · pp. 5–6

4.2 STOCHASTIC ANALYSIS WITH MOMENTUM ON THE BLENDED GRADIENT

  • One persistent momentum buffer on the blended gradient yields an O(T^-1/4) stochastic stationarity rate under a trajectory gradient bound.Its leading term has no explicit K dependence; lower-order terms retain linear K dependence under bounded variance, or become logarithmic under a stronger tail assumption.Source: §4.2 · pp. 6–7
  • The stochastic result assumes a trajectory gradient bound because the deterministic induction used to derive that bound does not apply to the stochastic one-step inequality.The specified step sizes also require ηRM ≤ 1/2.Source: §4.2 · p. 7

5 ITERATION-WISE INNER GAP

Entropy regularization and a weight floor let the method track a regularized inner target while retaining ergodic progress. Under a trajectory gradient bound, the analysis gives iteration-wise inner-gap control after burn-in.

5 ITERATION-WISE INNER GAP

  • Entropy regularization makes the inner objective strongly convex with a unique minimizer, while a positive weight floor keeps divergence to that target finite.The target is the minimizer of the regularized objective over the floored simplex; these choices do not enter the stationarity inequality.Source: §5 · p. 7
  • With the stated parameters and trajectory gradient bound, the inner gap is at most ϵ after burn-in T0 = Õ(M^4 ln²K/ϵ^4), while ergodic progress is retained.With β = ϵ/2, the bound holds for every t ≥ T0; the progress bound also holds for every T.Source: §5 · p. 7
  • The guarantee concerns the inner gap, not a competitive exponent for CA distance; conversion to that metric gives Õ(ϵ^-19) under the paper's comparison.The result is conditional on the trajectory gradient bound, unlike the deterministic rate theorem.Source: §5 · pp. 7–8

6 EXPERIMENTS

The experiments compare the method with multi-task baselines on dense-prediction and MultiMNIST tasks, then examine oracle, momentum, and weighting choices. Results are descriptive and use small task counts.

6 EXPERIMENTS

  • The dense-prediction results use one seed and average the final ten epochs, so they do not measure variation across independent runs.The method’s step sizes and schedule were tuned on the reported split, while baselines use reference hyperparameters.Source: §6 · p. 8
  • On Cityscapes, the method has the highest segmentation scores and lowest absolute depth error, while MOON has lower aggregate degradation; on NYU-v2, the method has the lowest aggregate degradation.Cityscapes degradation is −5.42% for MOON versus −3.38% for the method; NYU-v2 degradation is −6.80% for the method versus −3.72% for MOON.Source: §6 · p. 8, Table 3 · p. 9
  • With K = 2 on Cityscapes and K = 3 on NYU-v2, the dense-prediction experiments do not establish scaling with the number of objectives or isolate geometry and weighting contributions.The reported comparisons are descriptive single-run results.Source: §6 · p. 8

6.1 MULTIMNIST: COMPARISON WITH MOON AND ABLATIONS

  • 90.54% mean accuracy for the method versus 87.47% for MOON on regenerated MultiMNIST data; every tested η and κ grid value reaches at least 89.5%.The fixed validation split was drawn from MNIST-train and frozen before MNIST-test evaluation; only within-table comparisons are meaningful because MOON’s dataset instance was unavailable.Source: §6.1 · p. 8, Table 4 · p. 9
  • Removing momentum reduces accuracy by 8.86 points, while entropic and projected weights yield similar means at K = 2.These are fixed-hyperparameter substitutions rather than separately tuned comparisons; the oracle is best under the ablation’s fixed step sizes.Source: §6.1 · p. 9, Table 4 · p. 9

7 CONCLUSION

The conclusion summarizes the method’s deterministic and stochastic guarantees, regret bound, and post-burn-in inner-gap control, while noting important scope limits. The paper also reports its AI-use and reproducibility practices and records preliminary norm-pair facts used in proofs.

7 CONCLUSION

  • The single-loop entropic-LMO method gives O(ϵ^-2) deterministic stationarity with a derived gradient bound and O(ϵ^-4) stochastic stationarity with one persistent buffer and no explicit K in the leading term.It also establishes static regret on bounded gradient sequences and, with regularization and clipping, iteration-wise inner-gap control after burn-in.Source: §7 · p. 9
  • The selection-independent inner gap avoids minimizer-selection dependence in spectral and sign geometries, but does not itself certify simultaneous descent.The distinctive tracking result is an every-iterate guarantee after burn-in, not average-gap convergence alone.Source: §7 · p. 9
  • The deterministic trajectory gradient bound is derived only for the unregularized case; fixed relative oracle error is not yet covered for stochastic or iteration-wise guarantees.Experiments use K ≤ 3 and τ = κ = 0, so they do not test many-objective scaling or regularized tracking; tracking exponents are not competitive with Euclidean CA-distance results.Source: §7 · p. 9

AI USE STATEMENT

  • The authors report using a language model for editing, proof checking, experiment-code drafting, and related-work suggestions, and state that they verified its output.They attribute the research ideas, method, and main results to the authors, who take responsibility for the paper.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The paper states that assumptions and proofs are organized across appendices, with explicit constants, and that Algorithms 1–3 and the closed-form update specify the method.Appendix I documents experiment protocols and reports an anonymous implementation of the methods and experiments.Source: REPRODUCIBILITY STATEMENT · p. 10

A PRELIMINARIES FOR THE PROOFS

  • For the weight objective ψ(w) = ∥Aw∥*, every oracle output supplies a valid subgradient, so the entropic update’s sign follows from descent on that objective.The preliminary facts define A as the map blending task gradients and A⊤ as its adjoint; the mirror-descent bound needs no oracle selection rule.Source: A PRELIMINARIES FOR THE PROOFS · p. 13

B STRUCTURAL PROPERTIES OF THE CA DIRECTION

The section characterizes when the optimal blended gradient is unique and relates fixed scalarizations to Pareto stationarity. It also distinguishes the oracle’s worst-case improvement guarantee from simultaneous per-task descent.

B STRUCTURAL PROPERTIES OF THE CA DIRECTION

  • A strictly convex dual unit ball makes the optimal blended gradient unique across all minimizing weights; without strict convexity, an instance can have every weight optimal while blended gradients span a segment.Examples include ℓ1 gradients (1, 0) and (0, 1), and nuclear-norm matrices E11 and E22.Source: B STRUCTURAL PROPERTIES OF THE CA… · pp. 13–14
  • A first-order stationary point of any fixed scalarization is Pareto stationary, since that scalarization’s weight gives a zero blended gradient and hence a zero minimum norm.The passage notes that a minimizer is a special case of a first-order stationary point.Source: B STRUCTURAL PROPERTIES OF THE CA… · p. 14
  • An oracle output guarantees every objective improves by at least R∥G∥* only when the conflict-avoidance condition holds; a nonsmooth dual norm can admit oracle selections that violate it.When the dual norm is differentiable at G, the oracle is a singleton and the condition holds automatically for any minimizing weight.Source: B STRUCTURAL PROPERTIES OF THE CA… · p. 14
  • In the ℓ1 example, one valid oracle selection harms the second objective, whereas another improves both, even though both minimize the oracle objective.The convergence-rate analysis does not require the conflict-avoidance condition: selection affects the per-iteration guarantee, not the average-case rates.Source: B STRUCTURAL PROPERTIES OF THE CA… · p. 14

C PROOFS FOR THE DETERMINISTIC SETTING

The deterministic proof establishes convergence for general norms, then connects the blended-gradient residual to inner optimality measures. The stochastic proofs control weight movement and tracking error without a taskwise noise maximum in the leading term.

C PROOFS FOR THE DETERMINISTIC SETTING

  • T = ⌈(4∆L + M^2 ln K)/ϵ^2⌉ iterations suffice; every step uses only Assumptions 1–2 and identity (2), with no norm property beyond being a norm.Source: C PROOFS FOR THE DETERMINISTIC SETTING · p. 16
  • Average inner gap and CA distance converge whenever the actual blended-gradient residual converges in average, including for a fixed scalarization on its own convergent trajectory.The CA bound holds for every measurable choice of an optimal weighting, even when the optimal blended gradient is nonunique.Source: C PROOFS FOR THE DETERMINISTIC SETTING · pp. 16–17
  • Entropic weighting has static regret against the best fixed weighting in hindsight on arbitrary bounded matrix sequences, without requiring outer progress.The sequence may depend on preceding iterates.Source: C PROOFS FOR THE DETERMINISTIC SETTING · p. 17

D.1 THE WEIGHT INCREMENT

  • The stochastic weight-increment bound uses a weight-averaged noise term, leaving the number of objectives only in a second-order term.The stated noise condition does not require independence across tasks.Source: §D.1 · pp. 18–19

D.2 TRACKING ERROR

  • The tracking-error analysis uses no maximal inequality over tasks, so K enters only through the second-order term of the weight-increment bound.The pathwise drift from parameter movement is also independent of K.Source: §D.2 · p. 20

D.3 ONE-STEP INEQUALITY AND THE RATE

The stochastic rate proof combines an oracle evaluated using the previous momentum buffer with entropic weight updates. Under a trajectory gradient bound, it obtains an O(ϵ^-4) iteration count whose leading T^-1/4 term is independent of the number of objectives; the regret guarantee uses a fixed comparator in expectation.

D.3 ONE-STEP INEQUALITY AND THE RATE

  • The oracle uses the previous buffer so its direction is measurable before the current noise is sampled, allowing the relevant noise cross term to vanish in expectation.Evaluating the oracle at the current buffer instead would make the direction depend on current noise.Source: §D.3 · p. 21
  • O(ϵ^-4) iterations suffice for stochastic stationarity, and the leading T^-1/4 term has no explicit dependence on K.The two polynomial-in-K terms are lower order, of orders T^-3/4 and T^-1.Source: §D.3 · p. 22
  • Stochastic inner regret is bounded against the best fixed weighting in expectation, not the realized hindsight minimizer.The comparator must be deterministic for the noise cross term to vanish.Source: §D.3 · pp. 22–23
  • With conditional sub-Gaussian noise, the theorem's constants depend on K only through ln K under the stated horizon condition.The passage also states that almost surely bounded noise removes the K terms from those constants.Source: §D.3 · pp. 22–23
  • The stochastic rate assumes a trajectory gradient bound; the deterministic induction does not transfer because the stochastic inequality controls expected function values, not an almost-sure bound.An almost-sure bound on stochastic gradients would directly imply the required bound on expected gradients.Source: §D.3 · p. 23

E PER-TASK TRACKING VARIANT

The per-task momentum variant trades improved M-dependence for worse K-dependence than blended momentum. The iteration-wise inner-gap analysis uses a clipped entropic update and controls contraction toward a regularized, floored target with slack and moving-target sensitivity bounds.

E PER-TASK TRACKING VARIANT

  • Per-task tracking offers better M-dependence but worse K-dependence; only blended tracking has a leading term free of K under bounded variance alone.Whether the per-task variant's K dependence can be removed remains open.Source: E PER-TASK TRACKING VARIANT · p. 24

F.1 COMPUTING THE CLIPPED UPDATE

F.2 CONTRACTION WITH SLACK

  • The contraction argument for the inner objective incurs additive slack because it uses subgradient analysis for a potentially nonsmooth objective.The analysis does not establish contraction without slack; Euclidean squared-objective analyses rely on smoothness that need not hold for every dual norm.Source: §F.1 · p. 26, §F.2 · p. 27

F.3 THE MOVING TARGET

  • The moving-target sensitivity bound uses strong convexity and uniform closeness of the objectives, while stronger sensitivity estimates require additional regularity.The stated argument applies in every norm.Source: §F.3 · p. 27

F.4 THE RECURSION, THE BIAS, AND THE PROOF OF THEOREM 5.1

Theorem 5.1 tracks a regularized, floored target to obtain an iteration-wise inner-gap guarantee after burn-in while retaining ergodic progress. The guarantee is conditional on a trajectory gradient bound, which the paper does not derive for this setting.

F.4 THE RECURSION, THE BIAS, AND THE PROOF OF THEOREM 5.1

  • The regularized target's bias relative to the original inner problem is bounded by 2MKκ + τ ln K.The bound assumes a trajectory gradient bound and κ ≤ 1/K.Source: §F.4 · p. 28
  • After burn-in, the tracked weights have a controlled inner gap while the method retains ergodic stationarity; a tail-selected iterate carries both guarantees.The tracked target is the regularized floored minimizer, and the gap to the original inner problem is controlled through β.Source: §F.4 · p. 29
  • Theorem 5.1 assumes a trajectory gradient bound rather than deriving one; completing the proposed derivation would require checking remaining terms against ∆ and the horizon actually run.The authors state they have not carried out that verification.Source: §F.4 · p. 29

G UNIFORM CONVEXITY OF ℓp AND SCHATTEN-p NORMS

The appendix characterizes the uniform-convexity conditions for ℓp and Schatten-p norms, then records deterministic guarantees and limitations for inexact oracles. Its dense-task experiments use Cityscapes and NYU-v2, with hyperparameters selected on the reported evaluation splits.

G UNIFORM CONVEXITY OF ℓp AND SCHATTEN-p NORMS

  • For ℓp and Schatten-p norms, (S2) holds with q = 2 and μ = p−1 for 1 < p ≤ 2, and q = p and μ = 2^(1−p) for 2 ≤ p < ∞.For p > 2, the q = 2 condition fails; for p = 1, the unit ball is not strictly convex and no such inequality holds.Source: G UNIFORM CONVEXITY OF ℓp AND… · pp. 29–30
  • For p > 2, the power type q = p appears because the ℓp unit sphere is flatter than quadratic at coordinate points.The stated constant μ = 2^(1−p) is explicit but not sharp.Source: G UNIFORM CONVEXITY OF ℓp AND… · p. 30

H INEXACT ORACLES

  • A fixed relative oracle error is tolerated for deterministic stationarity and average inner optimality.The analysis does not extend this result to stochastic or iteration-wise guarantees.Source: H INEXACT ORACLES · p. 31
  • The five-step quintic Newton–Schulz implementation has no established uniform relative-error bound, leaving its connection to the oracle assumptions open.Extending the stochastic and iteration-wise guarantees to oracle error also remains open.Source: H INEXACT ORACLES · p. 31

I.1 CITYSCAPES AND NYU-V2

  • Cityscapes and NYU-v2 runs follow a 200-epoch protocol, with reported metrics averaged over the final 10 epochs from one seed.Those epochs are correlated and do not estimate variability across independent runs.Source: §I.1 · pp. 31–32
  • Dense-task hyperparameters are selected on the same validation or test split used to report results, without an additional held-out validation split.Cityscapes uses its 500-image validation split; NYU-v2 uses its 654-image evaluation (test) split.Source: §I.1 · p. 32

I.2 MULTIMNIST

On MultiMNIST, the study evaluates a spectral-oracle, entropic-weight, blended-momentum configuration against multi-task baselines and examines component choices. Near-uniform weights yield similar accuracy, so the reported baseline margin is not attributable to adaptive weighting at K = 2.

I.2 MULTIMNIST

  • MultiMNIST examples pair two MNIST digits for left- and right-digit classification, and all Table 4 methods use the same generated dataset and train for 100 epochs.The dataset contains 10,000 training and 1,000 test examples; images are resized to 28 × 43 and may overlap by up to 30%.Source: §I.2 · p. 32
  • The method forms a weighted aggregate of per-task gradients and computes its oracle direction from the previous momentum buffer, using spectral-norm blocks for matrix tensors and one Euclidean block for scalar and vector parameters.The implementation applies five Newton–Schulz steps to matrix tensors; the blended buffer starts at zero, with α0 = 1 and subsequent α = 0.1.Source: §I.2 · pp. 32–33
  • The comparison uses released MOON and Adam-based MGDA and FAMO implementations alongside Muon variants, but evaluates complete implementations rather than isolating one algorithmic component.MOON and the baselines differ in momentum conventions, weight decay, logit or task-weight updates, and auxiliary parameter updates.Source: §I.2 · p. 33
  • Hyperparameters are selected on a fixed 10% MNIST-train validation split, final candidates are evaluated over seeds 42, 43, and 44, and the frozen configuration is tested on MNIST-test.The selected configuration uses mean validation accuracy over epochs 96–100; Table 5 examines weight-step sensitivity.Source: §I.2 · p. 33, Table 5 · p. 33
  • The ablations are fixed-hyperparameter substitutions, so their oracle results do not establish superiority over a retuned Euclidean oracle, nor do their weight results establish equivalence of entropic and projected updates.The variants change one component while keeping other settings, including spectral-configuration step sizes, fixed.Source: §I.2 · p. 33
  • For κ ≤ 0.48, clipping results coincide with the unclipped run to reported precision.Table 6 reports test accuracy and final inner gap averaged over three seeds.Source: §I.2 · p. 33, Table 6 · p. 34
  • 91.11% mean accuracy at κ = 0.495 accompanies an empirical inner gap increase from 0.90 to 1.65; the similar accuracy means the baseline margin cannot be attributed to adaptive weighting at K = 2.As κ approaches 1/K = 0.5, the floor forces weights toward uniform.Source: §I.2 · p. 34
  • The inner-gap diagnostic uses per-task gradients on a fixed 256-example probe, but geometry-dependent gaps are not numerically comparable and are not the full-dataset inner optimization error.Main configurations use τ = κ = 0, so they do not directly test the regularized iteration-wise tracking guarantee.Source: §I.2 · p. 34

Related papers

Loading 2610.13773v1…