PaperlayerView exact revision on arXiv

Source-linked AI summary

Conda: Column-Normalized Adam for Training Large Language Models Faster

Junjie Wang, Pan Zhou, Yiming Dong, Huan Li, Jia Li, Xun Zhou, Qicheng Lao, Cong Fang, Zhouchen Lin

arXiv:2509.24218v2cs.LGcs.AI

TL;DR

LLM training is computationally expensive, and Adam updates can suffer from poor spectral conditioning while Muon lacks Adam’s coordinate-wise adaptivity. Conda combines orthogonal-subspace projection with column-wise second-moment normalization, and the paper reports faster convergence than AdamW on LLaMA models.

  • Problem

    Adam’s updates can have poor spectral conditioning and low-rank structure, while Muon’s global normalization lacks Adam’s coordinate-wise adaptivity.

    Source: §1 · p. 2

  • Method

    Conda projects updates into an orthogonal subspace and applies separate second-moment normalization to each column using projected gradients.

    Source: §1 · p. 2

  • Results

    Conda consistently outperforms AdamW, Muon, and other baselines in pre-training; on LLaMA it achieves 2∼2.5× AdamW’s convergence speed in training steps and training time.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    Conda is presented as a promising optimizer for efficient large-scale LLM training, with ablations reporting robustness across diverse training conditions.

    Source: §5 · p. 10

    Conda has not yet been evaluated on models larger than 1B parameters or on Mixture-of-Experts architectures.

    Source: §5 · p. 10

Abstract

from arXiv · show

Large language models (LLMs) have demonstrated impressive generalization and emergent capabilities, yet their pre-training remains computationally expensive and sensitive to optimization dynamics. While Adam-based optimizers offer fast convergence by adapting learning rates coordinate-wise, recent studies reveal that their updates often suffer from poor spectral conditioning and low-rank structures, hindering efficiency. Muon addresses this issue via global spectral normalization but lacks the per-coordinate adaptivity of Adam. In this work, we propose Column-Normalized Adam (Conda), a novel optimizer that bridges the strengths of both approaches. Conda projects updates into an orthogonal subspace and applies column-wise second moment normalization based on the projected gradients, thereby achieving both improved spectral conditioning and maintaining coordinate-wise adaptivity. This design alleviates the spectral pathologies of Adam while preserving its fast convergence behavior. Extensive experiments on the LLaMA and GPT-2 series show that Conda consistently outperforms AdamW, Muon, and other baselines in pre-training. Remarkably, on the LLaMA series, Conda achieves 2-2.5 the convergence speed of AdamW, measured in both training steps and training time. Further ablations demonstrate its robustness under diverse training setups. These results collectively highlight Conda as an effective and broadly applicable optimizer for large-scale LLM training. The code is released on https://github.com/jie040109/Conda

1 INTRODUCTION

LLMs make optimization efficiency a critical bottleneck, while Adam’s adaptive updates and Muon’s spectral normalization each leave a challenge for efficient transformer training. Conda combines orthogonal-subspace projection with column-specific second-moment normalization, and experiments report faster convergence than AdamW.

1 INTRODUCTION

  • LLM pre-training costs make optimization efficiency a critical bottleneck, and gradient and Hessian heterogeneity makes uniform SGD update rules ineffective for transformers.Adam and AdamW address this by adapting coordinate-wise learning rates using gradient second-moment estimates.Source: §1 · pp. 1–2
  • Adam updates for transformer parameter matrices can have high condition numbers and low-rank structure, while Muon’s global spectral normalization lacks Adam’s coordinate-wise adaptivity.Muon’s uniform normalization may overshoot updates and neglect fine-grained gradient variations.Source: §1 · p. 2
  • Conda projects updates into an orthogonal subspace and normalizes each column using projected-gradient second moments, retaining Adam’s coordinate-wise adaptivity while alleviating its spectral pathologies.The design aims to preserve the update matrix’s structure and relative scaling for better-conditioned updates and more stable convergence.Source: §1 · p. 2
  • 2∼2.5× faster convergence than AdamW on the LLaMA series, measured in training steps and wall-clock time; Conda also shows gains on GPT-2 and fine-tuning tasks.Ablations across sequence length, hyperparameters, subspace update frequency, and memory usage confirm robustness and scalability.Source: §1 · p. 2

2 RELATED WORKS

Related work covers adaptive optimizers that modify per-parameter or layer-wise updates, as well as methods that exploit matrix structure. The latter include curvature approximations, memory-efficient factorizations, and spectral or subspace transformations.

2 RELATED WORKS

  • Adaptive methods span per-parameter scaling and momentum, memory-efficient or second-order variants, and Adam extensions using Nesterov acceleration.Examples include Adagrad, RMSprop, Adam, AdamW, Adan, Win, Lion, Sophia, and Adam-mini.Source: §2 · p. 2
  • KFAC and Shampoo use Kronecker-factored curvature approximations, Adafactor factorizes second moments, and LAMB applies layer-wise normalization.These methods exploit matrix structure or layer-level information for efficient or stable updates.Source: §2 · p. 2
  • Structural optimizers include GaLore’s low-rank gradient projection, SOAP’s eigenbasis Adam-style updates, Muon’s Newton–Schulz spectral regularization, and AdaDiag’s near-diagonal preconditioning.The cited work presents these approaches as evidence that structural awareness can enhance training efficiency at scale.Source: §2 · p. 3

3 COLUMN-NORMALIZED ADAM

Transformer updates can be spectrally ill-conditioned, while Muon’s uniform singular-value normalization may distort update strengths. Conda combines subspace projection with column-specific second-moment normalization to retain coordinate-wise adaptivity and better preserve update structure.

3.1 PRELIMINARY AND MOTIVATION

  • Both SGD-momentum and Adam updates for transformer matrices can have high condition numbers and near-low-rank structure, slowing parameter updates.This motivates combining Muon-style normalization with Adam’s faster convergence potential.Source: §3.1 · pp. 3–4

3.2 COLUMN-NORMALIZED ADAM

  • Muon scales even very small singular values to one, which can distort desired update strengths and produce overly aggressive steps in those subspaces.A smaller learning rate can protect those directions but hampers updates in subspaces associated with larger singular values.Source: §3.2 · p. 4
  • The section reformulates Muon using first and second moments, aligning its form with Adam and clarifying its use of uniform second-moment scaling and subspace projection.Muon’s second moment is formed from singular values, and its update is projected back through U_t.Source: §3.2 · p. 4
  • Conda projects Adam’s second-moment estimate into the subspace induced by the first moment, aligning their statistics for more stable and coherent updates.The vanilla Adam second moment does not necessarily reside in that subspace.Source: §3.2 · p. 5
  • Muon uses row-wise adaptive learning rates, whereas Conda retains Adam’s coordinate-wise adaptation within the projected subspace.Conda normalizes each column using its own second-moment entries, making normalization column-specific and milder while preserving the relative order of singular values.Source: §3.2 · pp. 5–6
  • Conda uses lazy SVD updates, computing the decomposition every 2,000 iterations rather than at every iteration.The section says this setting works well across all experiments.Source: §3.2 · p. 6

4 EXPERIMENTS

Conda outperforms optimizer baselines in LLaMA and GPT-2 pre-training and in LoRA fine-tuning across LLaMA scales. Ablations examine projection, sequence length, update frequency, hyperparameter sensitivity, and memory use.

4.1 LLM PRE-TRAINING

  • Conda beats all baselines across LLaMA 60M–1B and GPT-2 125M/355M; on LLaMA-1B it is 2.7× faster in steps and about 2.5× in time than AdamW.The LLaMA speed advantage over AdamW exceeds 2× across model sizes in both steps and training time.Source: §4.1 · p. 7, Figure 2 · p. 7, Figure 3 · p. 7

4.2 LLM FINE-TUNING

  • In LoRA fine-tuning, Conda reaches 78.8% average accuracy on LLaMA-7B and 84.1% on LLaMA3-8B, leading AdamW across the three evaluated scales.On LLaMA3.2-1B, Conda scores 67.0% versus AdamW’s 59.2%.Source: §4.2 · p. 8, Table 3 · p. 9

4.3 ABLATION STUDY

  • Removing subspace projection does not substantially hinder smaller models, but LLaMA-1B is more sensitive, underscoring the need to align second-moment estimates within the subspace.Table 4 compares perplexity with and without projection.Source: §4.3 · pp. 8–9, Table 4 · p. 10
  • At sequence length 1,024 on LLaMA-1B, Conda achieves lower validation loss than all baselines.Other training settings are held the same as in the pre-training experiments.Source: §4.3 · p. 9, Figure 4 · p. 9
  • Conda maintains stable perplexity across subspace update frequencies T from 2 to 20,000 steps; T = 2,000 is used for wall-clock efficiency and scalability.T = 500 or 1,000 appears to be a sweet spot in the figure.Source: §4.3 · p. 9, Figure 4 · p. 9
  • On LLaMA-130M, Conda achieves consistently low perplexity across the tested β1 and β2 configurations, indicating robustness to these hyperparameter choices.The learning-rate sensitivity results in Appendix A.2 are reported as similarly robust.Source: §4.3 · p. 9, Table 5 · p. 10
  • Conda adds less than 1–2 GB of peak GPU memory versus AdamW in most tested settings, which the authors characterize as practical for large-scale training.The comparison spans LLaMA models from 60M to 1B using practical GPU configurations.Source: §4.3 · p. 9, Table 6 · p. 10

5 CONCLUSION

Conda combines column-specific spectral normalization with Adam’s coordinate-wise adaptivity to address spectral inefficiencies in transformer optimization. The paper reports faster convergence and robustness across training conditions, while noting that larger models and MoE architectures remain unevaluated.

5 CONCLUSION

  • 2∼2.5× faster convergence than AdamW on LLaMA models, measured in both training steps and training time.The conclusion also reports extensive ablations across diverse training conditions.Source: §5 · p. 10
  • Conda has not yet been evaluated on models larger than 1B parameters or on Mixture-of-Experts architectures.The authors cite computational resource constraints and identify these settings as future work.Source: §5 · p. 10

A.1 MORE EXPERIMENTAL RESULTS

The appendix compares Conda with Sophia-G during GPT-2 pre-training. Conda achieves lower validation loss and faster convergence on both reported model sizes.

A.1 MORE EXPERIMENTAL RESULTS

  • The Sophia comparison uses the best-performing Sophia configuration reported in its original publication.The comparison is presented as a more comprehensive evaluation of Conda.Source: §A.1 · p. 16
  • Conda achieves lower validation loss and faster convergence than AdamW and Sophia-G on GPT2-125M and GPT2-355M.Sophia-G shows slight improvements over AdamW but consistently lags behind Conda.Source: §A.1 · p. 16, Figure 5 · p. 16

A.2 MORE ABLATION STUDY

The ablations examine optimizer sensitivity to learning rates and subspace-update frequency. Conda remains stable at higher learning rates, while update frequencies T ≥100 add negligible overhead and performance varies little across several larger T values.

A.2 MORE ABLATION STUDY

  • Only Conda trains stably and converges rapidly at the relatively large learning rates tested on LLaMA-130M; Muon and Adam show loss spikes and fail to converge.The comparison reports final validation perplexity across learning rates.Source: §A.2 · p. 16, Table 7 · p. 17
  • A smaller-learning-rate sweep found that Muon and Adam perform best at a learning rate of 1e−3 on LLaMA models from 60M to 350M parameters.The sweep procedure is detailed in Appendix A.5.Source: §A.2 · p. 16
  • For T ≥100, the SVD step adds negligible overhead in the reported LLaMA-60M and LLaMA-130M timing experiments.Training time and validation perplexity across T are reported for LLaMA-60M and LLaMA-130M.Source: §A.2 · pp. 16–17, Table 8 · p. 17, Table 9 · p. 17
  • Performance differences among T ∈{500, 1000, 2000} are relatively small, so the main experiments use T = 2,000 for practical large-scale pre-training.The authors say this larger frequency reduces preconditioner-update frequency without sacrificing performance and supports robustness to T.Source: §A.2 · p. 17

A.3 PROOFS OF LEMMAS

The proofs recast Muon updates in an equivalent moment-estimation form and contrast that formulation with Conda’s column-wise update. They also specify an order of operations required to preserve equivalence.

A.3 PROOFS OF LEMMAS

  • Muon’s parameter update is reformulated as an equivalent expression, with diag(Σt) mapping singular values to a vector of dimension Rmin(m,n).The vector 1_n is defined as an n-dimensional vector whose entries are all ones.Source: §A.3 · p. 17
  • To preserve equivalence with O_t = U_tV⊤, the numerator matrix product must precede element-wise division, which must precede the remaining matrix multiplication.The passage warns that element-wise division and matrix multiplication are not interchangeable.Source: §A.3 · p. 18
  • The proof contrasts the reformulated Muon update with a Conda update expressed using columns of the update matrices.The supplied passages identify U_t,:i as the i-th column of U_t and N_t,i,j as an entry of N_t.Source: §A.3 · p. 18

A.4 PSEUDOCODE FOR CONDA

The pseudocode updates the momentum, periodically derives a projection subspace from its SVD, and uses projected gradients to normalize the update. It also selects the projection side according to the weight-matrix dimensions and applies a tuned scale factor.

A.4 PSEUDOCODE FOR CONDA

  • Conda periodically updates its projection subspace from the momentum matrix’s SVD, reusing the prior subspace between updates.The algorithm specifies a subspace update frequency T and returns the updated weight matrix after the convergence criterion is met.Source: §A.4 · p. 19
  • For weight matrices with m ≤ n, experiments favor left projection; when m > n, they favor right projection.The scale factor adjusts the relative learning rates for one- and two-dimensional parameters and is selected using validation performance.Source: §A.4 · p. 19
  • The update projects the momentum and gradient, forms a second-moment estimate from projected gradients, then normalizes and maps the result back.The pseudocode applies bias correction and adds epsilon to the square-root denominator before computing the update.Source: §A.4 · p. 20

A.5 DETAILED PRE-TRAINING SETTING

The section describes pre-training setups for LLaMA and GPT-2, including training schedules and optimizer hyperparameter selection. It notes that limited compute constrained the learning-rate search for the largest LLaMA model and several GPT-2 baselines.

A.5 DETAILED PRE-TRAINING SETTING

  • LLaMA models use a 256-token sequence length and 131K-token batch, with warmup, cosine decay, and size-dependent weight decay and gradient clipping.Conda uses one hyperparameter set from 60M to 1B parameters: learning rate 0.01, betas (0.9, 0.99), scale 0.25, and T = 2,000.Source: §A.5 · p. 21
  • Learning rates are selected by validation perplexity for LLaMA models, but compute limits prevent exhaustive tuning at 1B parameters.For the 1B model, each method reuses its 350M-model learning rate if training remains stable; optimizer beta settings are also specified in the text and Table 12.Source: §A.5 · p. 21, Table 12 · p. 22
  • GPT-2 Small and Medium are trained on OpenWebText for 100,000 steps with batch size 480, sequence length 1,024, cosine decay, and 2,000 warmup iterations.Because extensive searches were unavailable, Conda and other methods use learning rates scaled relative to AdamW; Conda’s rate is ten times AdamW’s on both GPT-2 sizes.Source: §A.5 · pp. 21–22, Table 14 · p. 22

A.6 DETAILED FINE-TUNING SETTING

The appendix specifies a LoRA fine-tuning evaluation and contrasts Conda’s design and implementation with SOAP. It attributes Conda’s stability and lower projection-update overhead to its projected-gradient estimator and momentum-based subspace construction.

A.6 DETAILED FINE-TUNING SETTING

  • LoRA fine-tuning compares Conda with AdamW on three LLaMA models trained on Commonsense170K and evaluated on commonsense reasoning benchmarks.The section states that the methods use identical LoRA settings.Source: §A.6 · p. 22, Table 15 · p. 23

A.7 CONCEPTUAL AND ALGORITHMIC DIFFERENCES FROM SOAP

  • The authors frame Conda’s design as integrating Adam’s coordinate-wise adaptivity into Muon’s spectrally normalized framework.Their reformulation of Muon is presented as a foundation for algorithmic improvements and as a conceptual connection to adaptive optimizers including Adam and Adafactor.Source: §A.7 · p. 23
  • Conda estimates second moments from projected gradients in the first-moment subspace; the native unaligned estimator caused instability and divergence in larger-model training.The authors report that projected-gradient estimates keep curvature estimation consistent with the update direction, stabilizing training.Source: §A.7 · p. 23
  • Unlike SOAP’s second-moment matrices and eigendecomposition, Conda obtains its projection subspace by applying SVD directly to the first-moment estimate.The authors state that momentum suppresses stochastic-gradient noise and yields a more accurate, stable subspace estimate.Source: §A.7 · pp. 23–24
  • Conda uses projection update frequency T = 2000, whereas SOAP uses T = 10, resulting in lower runtime overhead for Conda.The authors attribute this efficiency to momentum-based subspace definition, which allows infrequent updates without sacrificing performance.Source: §A.7 · p. 24

Related papers

Loading 2509.24218v2…