PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimizing Large Language Models with Chained LMOs

Sungyoon Kim, Kaan Ozkara, Youngsuk Park

arXiv:2610.10975v1cs.LGcs.AI

TL;DR

Muon variants that compose matrix normalizations lack a unified perspective, and many do not fit the standard LMO framework. The paper introduces chained LMOs, analyzes when chaining can help, and proposes TensorChain; in pretraining comparisons, TensorChain uses fewer tokens than Muon to reach the same validation loss.

  • Problem

    Muon has motivated multiple matrix-normalization optimizers, but these methods remain fragmented and lack a unified perspective.

    Source: §1 · p. 2

  • Method

    The paper represents Muon variants as chained LMOs, analyzes chaining in linear associative memory, and introduces TensorChain with normalization across layers.

    Source: §1 · p. 2, §6 · p. 10

  • Results

    Across four settings, TensorChain requires on average 9.6% fewer tokens than Muon and 7.7% fewer than MuonEq to reach the same validation loss.

    Source: §5.2 · p. 9

  • Takeaways & Limitations

    Under anisotropic embeddings and power-law pair frequencies, msign ◦RN can achieve asymptotically perfect retrieval while Muon and gradient descent fail.

    Source: §1 · p. 2

    The non-LMO characterization applies to exact matrix sign and stateless chains, and covers every Table 1 chain up to transpose.

    Source: §2.1 · p. 3

Abstract

from arXiv · show

Muon has motivated a growing family of optimizers that compose multiple matrix normalizations, but these methods remain fragmented and lack a unified perspective. We introduce chained linear minimization oracles (chained LMOs), which cast these methods as compositions of LMOs. Despite their empirical success, many chains fall outside the standard LMO framework and can diverge on smooth convex objectives. To explain why composition can nevertheless help, we turn to linear associative memory and show that chaining can improve over Muon under anisotropic embeddings. Empirically, we propose TensorChain, a novel optimizer within the framework that stacks compatible weight matrices across different layers and normalizes the 3d tensor across its axes. In Qwen3 0.6B and 1.7B pretraining, TensorChain outperforms all chained baselines in average token efficiency, with average token savings of 9.6% over Muon at matched validation loss.

1 INTRODUCTION

Muon variants that combine matrix normalizations share a composition-of-LMOs structure, which the paper formalizes as chained LMOs. The introduction motivates this framework with theoretical analysis, a learning advantage under anisotropy, and a new normalization axis for empirical comparison.

1 INTRODUCTION

  • Muon variants that apply row, column, or elementwise normalization around the matrix-sign step can be represented as compositions of LMOs.Muon itself is the single-LMO case over the spectral-norm ball.Source: §1 · pp. 1–2
  • The paper introduces chained LMOs as a common representation of Muon variants and evaluates them under a unified experimental protocol.The controlled comparison is motivated by the methods’ originally different evaluation setups.Source: §1 · p. 2
  • Many proposed chains fall outside the standard LMO framework, while some can diverge on smooth convex objectives; the paper also identifies chains that can help learning.The introduction frames linear associative memory as a complementary perspective for explaining a learning advantage from chaining.Source: §1 · p. 2
  • Under anisotropic embeddings and power-law pair frequencies, msign ◦RN can achieve asymptotically perfect retrieval while Muon and gradient descent fail.The result is presented as a provable learning advantage from combining different normalizations to address distinct sources of anisotropy.Source: §1 · p. 2
  • TensorChain adds normalization across the layer dimension and outperforms chained Muon baselines in the 1M-token batch regime, remaining competitive in the 4M-token regime.The introduction describes a common training setup for this comparison.Source: §1 · p. 2

2 OPTIMIZING WITH CHAINED LMOS

The paper tests whether compositions of normalization maps remain LMOs and analyzes what this means for optimization guarantees. Many practical chains are not LMOs of any fixed set, and their convergence behavior ranges from divergence to dimension-dependent guarantees.

2.1 CHAINED LMOS ARE NOT NECESSARILY LMOS

  • Many practical normalization chains are not LMOs of any fixed constraint set, although some compositions remain in the LMO class.For example, positive row scaling preserves entrywise signs, so Sgn ◦RN = Sgn.Source: §2.1 · p. 3
  • The non-LMO characterization covers exact matrix sign and stateless chains, including every chain in Table 1 up to transpose.The theorem’s stated shape and zero-row/column conditions are part of its scope.Source: §2.1 · p. 3

2.2 CONVERGENCE BEHAVIOR OF CHAINED LMOS

  • A positively aligned chained update can make optimization diverge even on a smooth convex objective.The paper constructs a convex, 1-smooth quadratic counterexample for a positively scale-invariant map with positive alignment on some input.Source: §2.2 · p. 4
  • Several chains containing both msign and Sgn, as well as msign ◦E in sufficiently large shapes, lack convergence guarantees even over smooth functions.The length-4 Aurora chain also admits a positively aligned input for n = 2 and m ≥258.Source: §2.2 · p. 4
  • For negatively aligned chains, approximate-LMO analysis gives convergence guarantees, but its rates lose a factor of 1/ν relative to exact-LMO rates.The framework applies to the stateless versions of NorMuon, Muon+ with one normalization, and MuonEq; msign ◦RN and RN ◦msign have approximation factor 1/√m.Source: §2.2 · p. 4
  • The approximation bounds for −RN ◦msign and −msign ◦RN tend to zero with dimension, so guarantees derived from them degrade as dimension grows.The stated bounds are ν ≤3(m −1)−1/4 for −RN ◦msign when m is odd, and ν ≤3/√m −1 for −msign ◦RN.Source: §2.2 · p. 4

3 PROVABLE BENEFIT OF CHAINING IN LINEAR ASSOCIATIVE MEMORY LEARNING

The paper studies when chaining helps in linear associative memory, where Muon can be insufficient if embeddings add a second source of anisotropy. Under stated scaling and anisotropy assumptions, row normalization followed by the matrix sign achieves asymptotically perfect one-step retrieval accuracy while gradient descent and Muon fail.

3 PROVABLE BENEFIT OF CHAINING IN LINEAR ASSOCIATIVE MEMORY LEARNING

  • With anisotropic embeddings, Muon may not suffice; msign ◦RN can help where Muon’s whitening of label-frequency anisotropy alone does not address the second source.The paper relates anisotropic embeddings to outlier coordinates in trained-transformer hidden representations.Source: §3 · p. 5
  • The analysis compares gradient descent, Muon, and msign ◦RN after one update from zero in linear associative memory.The chain is tailored to the setting’s anisotropy, and the paper notes that other anisotropy patterns may favor other chains.Source: §3 · p. 5
  • Row normalization removes answer-coordinate scales before the matrix-sign step, allowing useful contributions to accumulate across coordinates when their scale sum diverges.The paper contrasts this with GD and Muon, whose relevant quadratic and quartic scale sums remain bounded in the stated regime.Source: §3 · pp. 5–6
  • Under Assumptions 1 and 2, chained Muon’s asymptotic one-step accuracy tends to 1, while gradient descent’s and Muon’s tend to 0.The result concerns the limit in probability as n →∞.Source: §3 · p. 5

4 EXPLORING THE SPACE OF CHAINS

The section explores new chained-normalization designs, including layer-axis normalization, long-side normalization, and replacing the spectral step with row–column rescaling. TensorChain is formulated as a chained LMO when idealized, while tested Sinkhorn variants do not outperform Muon.

4.1 TENSORCHAIN: NORMALIZING ACROSS LAYERS

  • TensorChain stacks same-shape, same-role parameter momenta across layers and normalizes tensor fibers along the layer axis to prevent individual layers dominating corresponding update coordinates.Its fiber normalizations are exact LMOs over product ℓ2 balls, and slice-wise matrix sign is an exact LMO over product spectral-norm balls.Source: §4.1 · p. 6
  • Idealized TensorChain composes exact LMOs over product norm balls, but its resulting map is not an LMO of any fixed set when m, n ≥5.It also admits a negative-alignment input and can diverge on a convex smooth objective under non-summable step sizes.Source: §4.1 · p. 6

4.2 AURORA-LN: NORMALIZING ALONG THE LONG SIDE

  • Aurora-LN removes Aurora’s accumulated diagonal row-preconditioner state and applies the chain along each matrix’s long side, transposing wide matrices for the operation.The paper revisits this as a controlled simplification, not a novel optimizer or algorithmic primitive.Source: §4.2 · p. 7

4.3 REPLACING NEWTON–SCHULZ WITH SINKHORN NORMALIZATION

  • SinkhornChain alternates ℓ2 row and column rescaling instead of using the spectral step; for m > n, each sweep costs O(mn) FLOPs versus O(mn2) per Newton–Schulz step.The comparison is for a fixed number of iterations.Source: §4.3 · p. 7
  • None of the tested normalization variants or layerwise replacement configurations outperformed Muon; the strongest replaces NS only on Q, K and reaches validation loss 2.5945 versus Muon’s 2.5905.The authors report the exploration partly because Muon’s computational cost may be problematic at large scale.Source: §4.3 · p. 7

5 EXPERIMENTS

The experiments compare chained optimizers on Qwen3 pretraining and ablate TensorChain’s grouping and normalization choices. TensorChain improves token efficiency over Muon and MuonEq across the reported settings, while cross-layer normalization and layer grouping distinguish stronger configurations.

5.1 SETUP

  • Experiments train Qwen3-0.6B and 1.7B on FineWeb with sequence length 4096 and evaluate held-out FineWeb loss; most baselines share optima at 3 × 10^-3 for WSD and 12 × 10^-3 for cosine.The main corpus contains 30B tokens; additional 112B-token experiments without repeated tokens show similar results and consistent relative performance among the best optimizers.Source: §5.1 · pp. 7–8
  • The comparison uses selected baselines at shared learning rates, with AdaMuon excluded because its cosine optimum shifts substantially and its performance improves markedly there.TensorChain uses grouped weights with the same role and k = 5; the compared matrix optimizers use Nesterov momentum.Source: §5.1 · p. 8

5.2 RESULTS

  • Aurora-LN consistently improves over Aurora and is best in the largest cosine setting, while original Aurora is only slightly better than Muon.Figure 1 compares TensorChain with Muon and MuonEq, and Aurora-LN with Aurora.Source: Figure 1 · p. 8, §5.2 · p. 9
  • 9.6% fewer tokens than Muon and 7.7% fewer than MuonEq on average; TensorChain improves across model scales, batch sizes, and schedules among variants with comparable computational and state complexity.Table 3 estimates token reductions at a common validation-loss target by interpolation between adjacent checkpoints.Source: Table 3 · p. 9, §5.2 · p. 9

5.3 TENSORCHAIN ABLATIONS

  • Consecutive grouping is worse by approximately 0.007 validation loss under WSD and 0.01 under cosine; stride, greedy load-balanced, and all-compatible-layer groupings perform similarly.The authors interpret this as evidence that mixing matrices across distant layers matters.Source: §5.3 · p. 9, Table 4 · p. 10
  • 0.0208–0.0220 lower validation loss under WSD and 0.0248–0.0250 under cosine: TensorChain outperforms Sink–NS–Sink, which normalizes within matrices without stacking parameters across layers.The ablation uses Qwen3-0.6B, a 1M-token batch, and a 1× Chinchilla budget.Source: §5.3 · p. 10, Table 5 · p. 10
  • Increasing normalization iterations from k = 1 to 5 does not consistently improve validation loss; k = 1 is lowest or tied-lowest in every setting.This suggests the gains arise primarily from cross-layer normalization rather than convergence of the normalization procedure.Source: §5.3 · p. 10, Table 5 · p. 10

6 CONCLUSION

The paper frames chained LMOs as a unified account of Muon variants, argues that chaining can aid associative-memory learning despite limits of standard LMO guarantees, and reports TensorChain’s language-model pretraining results. It identifies open questions about scale, training settings, anisotropy, and additional tensor axes.

6 CONCLUSION

  • TensorChain adds normalization across the layer axis and can outperform existing chained variants in language-model pretraining.The conclusion highlights particularly strong gains at 0.6B as a question for future investigation.Source: §6 · p. 10
  • Future work should examine how TensorChain’s gains vary across model scales and training settings, extend associative-memory analysis to broader chains and anisotropy, and study additional axes in attention and mixture-of-experts models.The conclusion specifically notes particularly strong improvements at 0.6B.Source: §6 · p. 10
  • The authors disclose AI assistance with proof verification, novel-proof development, coding, experimental pipelines, and retrieval and organization of results.Source: AI USE STATEMENT · p. 11

A.1 FRAMEWORKS AROUND MUON AND VARIANTS

The discussion situates chained Muon variants among optimizer frameworks based on update geometry, explains how variants can be represented as normalization chains, and reviews associative-memory analyses of Muon. It also sets out the proof conventions used to assess whether chains remain LMOs and reports counterexamples to convergence guarantees for several chains.

A.1 FRAMEWORKS AROUND MUON AND VARIANTS

  • Standard frameworks use one geometry to determine both the update and its analysis, but composition generally breaks LMO closure and may prevent the usual oracle- or Lyapunov-based convergence arguments from transferring.Positively aligned chains can recover convergence through an approximate-LMO framework at an unavoidable dimension-dependent cost.Source: §A.1 · p. 14

A.2 EXISTING MUON VARIANTS AS CHAINED LMOS

  • Muon variants place normalization before or after matrix orthogonalization, while switching off exponential moving averages isolates their normalization chains without equating them to the full stateful optimizers.The passage gives reductions including NorMuon to RN ◦msign and Muon2 to msign ◦Sgn.Source: §A.2 · p. 15

A.3 LINEAR ASSOCIATIVE MEMORY AND MUON

  • In linear associative memory, imbalanced pair frequencies favor Muon over gradient descent because Muon whitens frequencies; with orthonormal embeddings, msign weights pairs equally while gradient descent learns proportionally to frequency.The cited analyses cover orthonormal embeddings and, in Kim et al. (2026), Gaussian embeddings.Source: §A.3 · p. 15

B APPENDIX: PROOFS IN SECTION 2

  • The appendix writes a chain as normalization maps N and defines the signed update T = −N; for the considered norm balls and odd selections, each constituent LMO is −N_j.This sign convention lets the non-LMO proofs test whether N is a linear maximization oracle and interpret conclusions for T as a minimization oracle.Source: B APPENDIX: PROOFS IN SECTION 2 · p. 16

B.1 NON-LMO RESULTS

This section analyzes when chained matrix and row/column normalizations do not define fixed-set linear maximization oracles, when such updates can diverge, and when specific chains satisfy approximate-LMO guarantees. It develops the appendix proofs using Gaussian-query and high-probability regularity assumptions, and establishes a population retrieval margin for bulk indices on EY with accompanying bounds.

B.1 NON-LMO RESULTS

  • Equal-norm non-idempotence rules out an LMO representation, and tall-matrix witnesses establish this obstruction for the listed normalization maps and compositions.The criterion applies when N(N(G)) differs from N(G) while their Frobenius norms agree.Source: §B.1 · pp. 16–17
  • Every finite composition containing both row normalization RN and column normalization CN is not a linear maximization oracle of any fixed set.The proof uses a positive witness family whose output norms remain fixed under iteration.Source: §B.1 · pp. 17–18
  • Positive matrices in the same positive diagonal orbit are unique when their squared row and column sums match, while limiting equilibration E is not a fixed-set LMO.A balanced-manifold perturbation produces another output with a larger inner product than E(G).Source: §B.1 · pp. 18–19
  • The odd composition msign ◦ E has a negative-alignment witness and therefore is not a linear maximization oracle of any fixed set.The passages construct witnesses across matrix shapes, including m,n ≥ 5.Source: §B.1 · pp. 19–20
  • Every finite composition containing both matrix sign msign and entrywise sign Sgn admits negative alignment and is not a fixed-set LMO.The argument covers rectangular matrices by zero-padding square witnesses.Source: §B.1 · pp. 20–21
  • For m,n ≥ 5, Theorem B.9 rules out fixed-set LMO representations for several listed matrix-sign and row/column-normalization compositions.The listed families include mixed row/column chains and alternating matrix-sign and row-normalization chains.Source: §B.1 · p. 21
  • Every listed normalization map and every finite composition containing both msign and Sgn has a signed update that is not an LMO of any fixed set.Source: §B.1 · p. 21

B.2 CONVERGENCE AND NON-CONVERGENCE GUARANTEES

  • Stateless updates using positively scale-invariant signed maps with positive alignment diverge on a convex, 1-smooth quadratic; the appendix also gives a negative-alignment example for two Aurora cycles.The Aurora example uses (msign ◦ RN)◦2 with n = 2 and m ≥258.Source: §B.2 · pp. 21–22
  • −(RN ◦ msign) is a 1/√m-approximate LMO of the row-unit ball, while −(msign ◦ RN) is one of the spectral-norm unit ball; transposition gives column-normalized analogues at 1/√n.Source: §B.2 · p. 24
  • For n = 2, the best uniform approximation factors for the two row-normalized chains vanish as m grows.Source: §B.2 · pp. 24–25

C APPENDIX: PROOFS IN SECTION 3

  • The appendix uses independent centered Gaussian query and response matrices and exact updates WG = G, WM = pol(G), and WC = pol(RN(G)).Here pol is the exact canonical polar factor, equivalently msign; the statements concern these exact updates, not a fixed number of Newton–Schulz iterations.Source: C APPENDIX: PROOFS IN SECTION 3 · pp. 25–26

C.1 QUERY-SIDE EVENT AND POPULATION GEOMETRY

  • The query-side lemma establishes a high-probability regularity event with inequalities holding simultaneously for all B and η.Source: §C.1 · p. 26
  • On EY, each leverage vector tj=(K⊤K)−1/2K⊤ej has Euclidean norm at most one.Source: §C.1 · p. 28
  • On EY, the response matrices commute with K⊤K, have bounded spectral quantities, and the empirical covariance and transformed-product errors are O(e) in operator norm.The spectral bounds include c≤cj≤C and c≤hj/√cj≤1.Source: §C.1 · pp. 28–29
  • The relative square-root perturbation estimate uses the Frobenius norm of the relative perturbation rather than a condition-number-dependent additive bound.Source: §C.1 · p. 29
  • For fixed b>0, population retrieval-margin failure vanishes uniformly over i≥bN conditional on EY, with the correct-coordinate score at least cbh−o(h).The proof bounds wrong deleted responses using Gaussian tails and transfers scores to the full response operator with an o(h) error.Source: §C.1 · pp. 29–30

C.2 CONDITION ON Y : RANDOMNESS REVEALED THROUGH G

This section conditions on Y to analyze the randomness revealed through G. Its proof develops conditional estimates and mean approximations for gradient descent, Muon, and the chained update.

C.2 CONDITION ON Y : RANDOMNESS REVEALED THROUGH G

  • Conditional on Y, the vectors ξ1, . . . , ξd are independent Gaussians, and their conditional law does not depend on Y.The proof uses this structure to establish estimates uniform in Y.Source: §C.2 · p. 31
  • The observed-gradient estimates control the relevant conditional quantities, with the truncation bias negligible when n/d → 0.The concentration argument uses matrix Bernstein and yields bounds with conditional probability tending to one uniformly in Y.Source: §C.2 · pp. 31, 34
  • The conditional mean is shown to be correct for gradient descent and Muon, with the corresponding bounds holding uniformly over Y.The proof separately establishes the gradient-descent result and the Muon result.Source: §C.2 · pp. 32–33
  • For the chained update, answer-coordinate scales cancel in RN(G) = RN(XK), and a fixed-Y approximation supports the chain’s conditional-mean analysis.The proof also controls a relative polar perturbation and obtains a row bound.Source: §C.2 · pp. 34–35

C.3 CONDITION ON (Y, G): RESIDUAL SCORE RANDOMNESS

After conditioning on (Y, G), the remaining randomness comes from the part of X not revealed by G, yielding a conditional law for retrieval scores. Under the theorem’s stated assumptions, the chain recovers a fixed query with probability tending to one, whereas gradient descent does not.

C.3 CONDITION ON (Y, G): RESIDUAL SCORE RANDOMNESS

  • Conditional on (Y, G), the remaining randomness is the component of X not revealed by G, and the retrieval scores have an exact conditional law.The centered scores are Gaussian under this conditional law.Source: §C.3 · p. 36
  • The theorem compares WC, WM, and WG on the same realized dataset.The supplied passage identifies the three accuracies but does not state their comparison values.Source: §C.3 · p. 37
  • Under the theorem’s assumptions, P{i is not recovered by WC} → 0, while P{i is recovered by WG} → 0.These limits are stated for a fixed deterministic query index; the WC bound is uniform over such indices.Source: §C.3 · pp. 37–39
  • The chain’s recovery bound is uniform over deterministic query indices, not a claim that one union bound recovers every query.The text states that this uniform bound suffices for the accuracy conclusion.Source: §C.3 · p. 39

D APPENDIX: PROOFS IN SECTION 4

The appendix verifies TensorChain’s chained-LMO representation and shows that the composition is not itself an LMO of a fixed set. It also gives a smooth convex counterexample where TensorChain optimization can diverge.

D APPENDIX: PROOFS IN SECTION 4

  • Each fiber-normalization stage is an exact LMO, and slice-wise matrix sign is also an exact LMO, so their composition is a chained LMO.Fiber normalization acts independently on nonzero fibers; matrix sign acts independently on each matrix slice.Source: D APPENDIX: PROOFS IN SECTION 4 · p. 40
  • The proof tests whether the composition is an LMO by embedding a matrix as the first tensor slice and zeroing the other slices.Row- and column-axis normalization preserve the nonzero slice’s signs and zero pattern before layer-axis normalization.Source: D APPENDIX: PROOFS IN SECTION 4 · p. 40
  • TensorChain admits a negative-alignment input, so it is not an LMO of any fixed set and can diverge on a smooth convex objective.The passage points to Proposition 1 for the smooth counterexample.Source: D APPENDIX: PROOFS IN SECTION 4 · p. 40

E APPENDIX: REPLACING NS WITH CHEAPER CHAINS

This appendix tests cheaper axis-wise normalizations as replacements for Newton–Schulz on Qwen3-0.6B. Selective replacements can approach Newton–Schulz, but none of the tested configurations matches Muon.

E APPENDIX: REPLACING NS WITH CHEAPER CHAINS

  • The experiments train Qwen3-0.6B for 8.8B tokens and report the best validation loss over a matrix-learning-rate sweep.The setup uses sequence length 4096 and 8401 optimization steps.Source: E APPENDIX: REPLACING NS WITH CHEAPER… · p. 41
  • Among FFN replacements, row–column equilibration performs best, and five sweeps yield only a small improvement over one.The authors use one sweep for the subsequent weight-wise ablation.Source: E APPENDIX: REPLACING NS WITH CHEAPER… · p. 41
  • Validation loss is 2.5945 for the strongest tested configuration, replacing NS only on Q and K, versus 2.5905 for Muon.Replacing additional parameter classes consistently degrades performance, and no tested configuration outperforms Muon.Source: E APPENDIX: REPLACING NS WITH CHEAPER… · p. 41, Table 6 · p. 42
  • The tested replacements can approach Newton–Schulz when used selectively, but none matches Muon; further equilibration sweeps do not close the gap.The figure compares replacements in the FFN while retaining NS5 on attention, and replacements on selected parameter classes while retaining NS5 elsewhere.Source: E APPENDIX: REPLACING NS WITH CHEAPER… · p. 41, Figure 2 · p. 41

F APPENDIX: RAW EXPERIMENT RESULTS

The appendix records the full baseline sweeps, learning-rate tuning, and training and validation trajectories, documenting schedule-dependent differences in optimizer rankings and learning-rate sensitivity.

F APPENDIX: RAW EXPERIMENT RESULTS

  • The appendix provides the experimental record behind the main-text comparisons, including full baseline sweeps, tuning, trajectories, and schedule-dependent differences.Unless otherwise stated, the configuration follows Section 5.1.Source: F APPENDIX: RAW EXPERIMENT RESULTS · p. 43

F.1 FULL QWEN3-0.6B BASELINE COMPARISON

  • AdaMuon’s best validation loss is 2.8405 at 27e −3, worse than TensorChain’s optimal 2.8379.The original learning-rate grid was insufficient, so two additional AdaMuon runs were conducted.Source: §F.1 · p. 43

F.2 MUON LEARNING-RATE SWEEPS

  • Muon learning-rate sweeps cover Qwen3 0.6B and 1.7B, 1M- and 4M-token batches, and WSD or cosine schedules.The reported tables include WSD sweeps for both model sizes and cosine sweeps across the listed batch sizes.Source: Table 9 · p. 44, Table 10 · p. 44, Table 11 · p. 44

F.3 WSD TRAINING AND VALIDATION TRAJECTORIES

  • WSD results include final training and validation losses and trajectories for Qwen3-0.6B and Qwen3-1.7B.The figure captions include 0.6B at a 1M-token batch and 1× Chinchilla, and 1.7B at a 4M-token batch and 4× Chinchilla.Source: Table 15 · p. 45, Table 16 · p. 45, Figure 4 · p. 45

F.4 COSINE TRAINING AND VALIDATION TRAJECTORIES

  • Cosine results include final training and validation losses and trajectories across the reported Qwen3 model, batch, and training-budget settings.The trajectory figures include both model sizes and 1M- or 4M-token batches at 1× or 4× Chinchilla.Source: Table 17 · p. 46, Table 18 · p. 46, Figure 6 · p. 46

F.5 TENSORCHAIN GROUPING ABLATION

The appendix compares TensorChain grouping choices, defines a cosine-schedule token-efficiency comparison, and reports a separate Teon comparison in one Qwen3 pretraining setup.

F.5 TENSORCHAIN GROUPING ABLATION

  • Consecutive grouping performs worse near its optimum than stride, greedy, and all-layer grouping, suggesting cross-depth mixing matters more than the precise partition.The 28-layer grouping schemes differ in whether they use contiguous groups, interleaved depths, parameter-balanced groups, or all compatible matrices together.Source: §F.5 · p. 48, Table 19 · p. 48

F.6 COSINE TOKEN EFFICIENCY

  • Cosine token efficiency measures the percentage of training tokens saved by TensorChain to reach the same validation-loss target; positive values favor TensorChain.The common target is the worse of the two final validation losses, so both methods reach it within the observed horizon; WSD is excluded because terminal decay concentrates loss reduction near the end.Source: §F.6 · p. 48, Table 20 · p. 48

F.7 COMPARISON WITH THE TEON OPTIMIZER

  • At their best reported validation losses, TensorChain reached 2.8990, Muon 2.9201, and Teon 2.9222, with the latter two reported at learning rate 3e-3.The Teon comparison used Qwen3 0.6B pretraining with a 1M batch, 1× Chinchilla tokens, and a WSD schedule.Source: §F.6 · p. 49, §F.7 · p. 49

Related papers

Loading 2610.10975v1…