PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Muon Fine-tune Adam-Pretrained Models?

Xingyu Qu, Peigeng Huang, Samuel Horvath

arXiv:2605.10468v1cs.LG

TL;DR

Fine-tuning Adam-pretrained models with Muon suffers from optimizer mismatch, limiting Muon’s practical use despite its pretraining efficiency. Through controlled experiments and LoRA studies across language and vision tasks, the paper shows that constraining updates reduces the gap, with LoRA-Muon matching or outperforming LoRA-Adam.

  • Problem

    Fine-tuning Adam-pretrained models with Muon remains underexplored, despite evidence that optimizer mismatch degrades performance and limits Muon’s practical applicability.

  • Method

    The paper combines controlled pretraining and fine-tuning experiments with analyses of implicit bias, update strength, LoRA rank, catastrophic forgetting, and LoRA variants.

  • Results

    Across language and vision tasks, LoRA-Muon matches or outperforms LoRA-Adam, whereas full fine-tuning exhibits an optimizer mismatch gap.

  • Takeaways & Limitations

    Constraining fine-tuning updates with LoRA can mitigate optimizer mismatch and support Muon use on Adam-pretrained models without performance loss.

  • Takeaways & Limitations

    The experiments use limited model scales and show that mismatch severity varies across tasks, while its theoretical characterization and determining factors remain open.

Abstract

from arXiv · show

Muon has emerged as an efficient alternative to Adam for pretraining, yet remains underused for fine-tuning. A key obstacle is that most open models are pretrained with Adam, and naively switching to Muon for fine-tuning leads to degraded performance due to an optimizer mismatch. We investigate this mismatch through controlled experiments and relate it to the distinct implicit biases of Adam and Muon. We provide evidence that the mismatch disrupts pretrained knowledge, and that this disruption scales with update strength. This leads us to hypothesize that constraining updates should mitigate the mismatch. We validate this with LoRA: across language and vision tasks, LoRA reduces the performance gap between Adam and Muon observed under full fine-tuning. Studies on LoRA rank, catastrophic forgetting, and LoRA variants further confirm that mismatch severity correlates with update strength. These results shed light on how optimizer mismatch affects fine-tuning and how it can be mitigated. Our code is available at https://github.com/XingyuQu/muon-finetune.

1. Introduction

Muon offers roughly 2× compute efficiency over Adam for pretraining but remains difficult to apply when fine-tuning Adam-pretrained models because optimizer mismatch degrades performance. The paper links this degradation to disrupted pretrained knowledge and shows that constraining updates with LoRA mitigates the gap across language and vision tasks.

  • Background: Muon achieves approximately 2× compute efficiency over Adam for large language model pretraining while requiring less memory by eliminating the second moment.Muon orthogonalizes the momentum matrix before each update.
  • Optimizer mismatch: Fine-tuning an Adam-pretrained model with Muon yields suboptimal results compared with Adam, creating a practical mismatch because most open models use Adam for pretraining.The mismatch also occurs in the reverse direction when Adam fine-tunes Muon-pretrained models.
  • Analysis: The paper analyzes optimizer mismatch through controlled experiments and relates it to distinct Adam and Muon implicit biases that produce structurally different pretrained weights.The analysis provides evidence that mismatch degrades performance by disrupting pretrained knowledge.
  • Mitigation: LoRA constrains updates and enables LoRA-Muon to match or outperform LoRA-Adam across language and vision tasks.Rank studies, catastrophic forgetting measurements, and LoRA-variant experiments further support the mitigation hypothesis.

2. Preliminaries

This section introduces Muon and Adam as optimizers with fundamentally different preconditioning schemes, and presents LoRA as a low-rank, parameter-efficient fine-tuning method that preserves pretrained weights.

  • Muon: Muon targets matrix-shaped parameters, is typically paired with Adam for embeddings and biases, and follows the Liu et al. (2025) implementation here.Muon implementations vary slightly across frameworks; the paper uses Liu et al.’s implementation except in Section 3.
  • Muon: Muon orthogonalizes gradient updates so singular directions receive nearly uniform step sizes through matrix-level preconditioning.Newton-Schulz iteration approximates the nearest semi-orthogonal matrix, while Polar Express replaces fixed coefficients with adaptive ones.
  • Adam: Adam, dominant in large-language-model pretraining and fine-tuning, independently rescales each parameter using element-wise first- and second-moment estimates.Muon instead adapts step sizes across singular directions, establishing the optimizers’ fundamentally different implicit biases.
  • LoRA: LoRA freezes pretrained weights and represents updates with trainable low-rank matrices, substantially reducing trainable parameters and memory requirements.For rank r ≪ min(m, n), only A and B are updated while W0 remains frozen; default initialization makes ΔW = 0 initially.
  • LoRA: LoRA can underperform full fine-tuning because of its low-rank constraint, but this constraint may help preserve pretrained knowledge by making the model learn less and forget less.Several variants use initialization techniques intended to bring LoRA updates closer to full fine-tuning.

3. Analyzing Optimizer Mismatch

Controlled experiments show that switching between Adam and Muon across pretraining and fine-tuning consistently degrades performance, consistent with their fundamentally different implicit biases and resulting pretrained weight structures. LoRA mitigates this mismatch by constraining updates, reducing the performance gap across pretrained models.

  • Reproducing the mismatch: Matched fine-tuning consistently outperforms mismatched fine-tuning for both Adam- and Muon-pretrained models, revealing a symmetric optimizer incompatibility.The experiments compare Full-Muon, Full-Adam, LoRA-Muon, and LoRA-Adam after pretraining 561M-parameter NanoChat models with either optimizer.
  • Implicit bias: Adam and Muon impose different implicit biases: Adam favors minimum max-norm solutions, whereas Muon favors minimum spectral-norm solutions.Adam uses element-wise preconditioning, while Muon uses matrix-level preconditioning based on (MM ⊤)−1/2.
  • Impact on fine-tuning: Muon-trained weights exhibit notably higher stable rank, indicating structurally different pretrained weights that can make mismatched updates incompatible with learned knowledge.Learning-rate sweeps show that mismatch shifts the perplexity curve upward and leftward, producing worse best perplexity at a smaller optimal learning rate.
  • Mitigating mismatch: LoRA constrains updates by preserving pretrained weights and optimizing low-rank adapters, thereby reducing mismatch severity.The low-rank constraint limits update extent, aligning with the observation that LoRA learns less and forgets less.
  • Mitigating mismatch: 39% and 78%: LoRA shrinks the perplexity gap for Muon-pretrained and Adam-pretrained models, respectively.LoRA also narrows matched-versus-mismatched learning-rate curves, while LoRA-Muon converges faster than LoRA-Adam early on Adam-pretrained models.

4. Experiments

Across language and vision benchmarks, full fine-tuning with Muon underperforms or closely trails Adam, whereas LoRA largely eliminates the gap and often gives Muon an advantage. Meta-analysis, rank studies, forgetting measurements, and weight-space analyses support constrained updates as a mitigation for optimizer mismatch.

  • Language tasks: Full-fine-tuned Muon underperforms Adam on NLU and NLG tasks, while LoRA-Muon matches or surpasses LoRA-Adam across language benchmarks.On GLUE, LoRA-Muon slightly outperforms LoRA-Adam, and LoRA-Muon-PE achieves the highest average accuracy; on Llama 2-7B, LoRA-Muon matches Adam on math and outperforms it on code and commonsense reasoning.
  • Image classification: In vision, the full-fine-tuning Adam–Muon gap is small, but Muon and Muon-PE outperform Adam on average under LoRA.This extends LoRA’s mismatch-mitigation effect beyond language tasks.
  • Statistical significance: 0.72% (95% CI: [0.41, 1.04], p < 0.001) is the pooled Adam–Muon gap reduction under LoRA, compared with 0.83% (95% CI: [0.45, 1.20], p < 0.001) for Muon-PE.The random-effects meta-analysis aggregates tasks from Tables 2–4 and confirms statistically significant mitigation.
  • Rank dependence: At higher LoRA ranks, Muon degrades on MetaMath while Adam continues improving, but the methods remain comparable across ranks on Code-Feedback.The results indicate that low-rank constraints help when mismatch is pronounced, whereas higher-rank updates increasingly resemble full fine-tuning.
  • Forgetting and weight displacement: Full-Muon causes more forgetting than Full-Adam despite worse fine-tuning performance, while LoRA-Muon preserves pretrained knowledge better than Full-Muon.Forgetting under LoRA-Muon initially decreases with rank, narrowing its gap with LoRA-Adam until it nearly vanishes at r = 32–64; weight displacement further links mismatch severity to disruption.
  • Efficiency: LoRA-Muon is only 1.1–1.2× slower than LoRA-Adam on Llama 2-7B and 1.0–1.1× slower on CLIP, indicating modest per-step overhead.Full-fine-tuning timing is confounded by different distributed strategies.

5. Discussion

The discussion attributes Adam–Muon fine-tuning mismatch to distinct implicit biases that disrupt pretrained knowledge, while showing LoRA mitigates the issue. It recommends LoRA-Muon as a practical Adam alternative and identifies open theoretical and experimental limitations.

  • LoRA mitigates optimizer mismatch, enabling LoRA-Muon to match or outperform LoRA-Adam across language and vision tasks.The discussion links mismatch to distinct implicit biases and disruption of pretrained knowledge.
  • Practical Recommendations: Under LoRA, Muon can replace Adam without performance loss while saving 50% optimizer-state memory.
  • Practical Recommendations: Muon requires separate learning-rate tuning because its optimal learning rate often differs from Adam’s.The difference is reported in Figure 4.
  • Practical Recommendations: Moderate LoRA ranks balance expressiveness against mismatch severity, and Adam-optimized LoRA variants may not transfer directly to Muon.
  • Limitations and Future Work: A theoretical characterization of how mismatched implicit biases disrupt pretrained knowledge remains open.Specialized initialization or warmup may reduce the structural gap before fine-tuning.
  • Limitations and Future Work: Muon-pretrained experiments are limited to NanoChat (561M), while Adam-pretrained experiments mainly use 7B with preliminary 13B results.

Impact Statement · Appendix · A. Related Work

The paper situates Muon within prior work on its mechanism, efficiency, improvements, and theory, while identifying optimizer mismatch during fine-tuning as an unresolved problem addressed through update constraints such as LoRA. The accompanying impact statement and appendix document societal considerations, analyses, evaluations, spectral studies, and computational resources.

  • Impact Statement: The impact statement notes that the work may have societal consequences but does not identify any requiring specific emphasis.
  • Appendix: The appendix includes theoretical analysis of optimizer implicit bias and a subsection on how LoRA mitigates mismatch.
  • Appendix: The appendix reports studies covering natural language understanding and generation, image classification, LoRA rank, catastrophic forgetting, and LoRA variants.
  • Appendix: The appendix includes spectral analyses of model weights and LoRA matrices.
  • Appendix: The appendix documents the computational resources used in the work.
  • A. Related Work: Muon combines momentum-based SGD with per-layer Newton–Schulz orthogonalization, interpretable as steepest descent under the spectral norm.The orthogonalization projects the momentum matrix toward semi-orthogonal matrices and equalizes contributions across update directions.
  • A. Related Work: Prior work establishes Muon’s scalability, efficiency, algorithmic improvements, and theoretical foundations, but has focused almost exclusively on pretraining.Reported developments include lower memory use than Adam, approximately 2× compute efficiency in scaling-law experiments, improved normalization and distributed integration, and convergence analysis with O(1/ϵ4) iteration complexity.
  • A. Related Work: Fine-tuning across optimizers creates a poorly understood mismatch that degrades performance, while constraining updates with methods such as LoRA can mitigate it.This issue is especially consequential because most open models are pretrained with Adam; LoRA is also established as a memory- and storage-efficient parameter-efficient fine-tuning method, though its low-rank constraint can underperform full fine-tuning.

B. Muon Implementation

Muon implementations differ in scaling and momentum rules, and this study adopts the Moonlight variant with Nesterov momentum. The experiments compare standard Newton-Schulz orthogonalization with the adaptive Polar Express variant.

  • Implementation differences: Muon implementations vary in the scaling factor applied to the orthogonalized update and the momentum update rule.The section introduces these differences before specifying the implementation used in the experiments.
  • Scaling: The original Muon uses max(1, m/n), whereas Moonlight uses the dimension-symmetric max(m, n) scaling factor.The original factor depends on the ordering of m and n; the Moonlight factor is symmetric.
  • Momentum: The study adopts Moonlight’s classical momentum, Mt = βMt−1 + Gt, and employs Nesterov-style momentum in all experiments.Original Muon instead uses the EMA update Mt = βMt−1 + (1 − β)Gt, similar to Adam’s first moment.
  • Orthogonalization: The experiments evaluate standard Newton-Schulz Muon and Muon-PE, whose adaptive coefficients accelerate convergence to the orthogonal matrix.Standard Newton-Schulz uses fixed coefficients (a, b, c) = (3.4445, −4.7750, 2.0315), while Polar Express precomputes iteration-specific coefficients on [ℓ, 1].

C. NanoChat Experiment … D.2. Muon

The NanoChat experiments use matched Adam and Muon pretraining setups, while the theoretical analysis shows that their proxies have distinct implicit biases. SignGD selects minimum max-norm solutions, whereas Muon selects minimum spectral-norm solutions in the simplified regression problem.

  • C. NanoChat Experiment: NanoChat uses a GPT-style decoder-only Transformer with RoPE, QK-norm, ReLU2, approximately 561M parameters, and ∼11B FineWeb-Edu tokens.The model has depth 20, hidden dimension d = 1536, and 12 attention heads.
  • C. NanoChat Experiment: Both models are pretrained for 21,400 iterations with 524,288-token batches and sequence length 2048, using parameter-group-specific learning-rate scaling.Muon uses matrix learning rate 0.02, while Adam uses 1e-3 tuned for comparable CORE performance.
  • C. NanoChat Experiment: Muon and Adam achieve similar final pretraining performance, with Muon converging slightly faster.The comparison uses training loss, validation BPB, and the CORE metric against GPT-2 Large.
  • D. Theoretical Analysis: Implicit Bias of Optimizers: The theoretical analysis studies Muon and SignGD on a simplified underdetermined linear regression problem, using SignGD as a proxy for Adam.For theoretical clarity, it omits momentum and assumes exact orthogonalization for Muon.
  • D.1. Sign Gradient Descent: SignGD’s proof establishes convergence through a scalar recurrence, with the residual tending to zero under diminishing step sizes.The argument uses Lemma D.1 to show limt→∞dt = 0.
  • D.1. Sign Gradient Descent: SignGD converges to a solution satisfying W∗x = y that achieves the minimum max-norm among all solutions.The theorem assumes x ≠ 0 and step sizes satisfying Lemma D.1 conditions.
  • D.2. Muon: Muon converges to a solution satisfying W∗x = y that achieves the minimum spectral norm among all solutions.The proof uses exact orthogonalization of rank-1 gradients and Lemma D.1 to establish convergence.

D.3. LoRA Mitigates Mismatch: Theoretical Analysis

The analysis models fine-tuning as learning a correction that fits the pretrained model’s residual, revealing that SignGD/Adam and Muon select different exact-fit solutions. Under their native norm geometries, matched optimizers require smaller budgets and provide corresponding guarantees on preserving old-task behavior.

  • Correction view: Fine-tuning from W0 is equivalent to learning a correction matrix ∆ from zero that fits the residual r0.The analysis uses SignGD as an Adam proxy and idealized Muon without momentum or approximate orthogonalization.
  • Implicit bias: SignGD and idealized Muon initialized at W0 converge to different exact-fit correction solutions, with Muon selecting a minimum-Frobenius-norm correction.The Muon solution is characterized as µ ∈ arg min ∆: ∆z=r0 ∥∆∥2.
  • Budgeted fine-tuning: Matched optimizers reach exact fit with the smallest budget in their native geometries: max norm for Adam/SignGD and spectral norm for Muon.This establishes that optimizer mismatch can make an exact-fit correction infeasible at budgets where the matched correction is feasible.
  • Budgeted fine-tuning: Whenever either native-budget inequality is strict, a nonempty interval exists where the matched exact-fit correction is feasible but the mismatched correction is not.The result formalizes how native geometry constrains the severity of optimizer mismatch.
  • Old-task damage: For old-task inputs satisfying W0x = y, matched SignGD minimizes the max-norm damage bound, while matched Muon minimizes the spectral-norm damage bound among exact-fit solutions.The bounds derive from |(∆x)i| ≤ ∥∆∥max∥x∥1 and ∥∆x∥2 ≤ ∥∆∥2∥x∥2.

D.4. A Fixed-Subspace LoRA Surrogate … E.4. LoRA Rank Study

The appendix formalizes a fixed-subspace LoRA model for analyzing optimizer mismatch and specifies the experimental protocols for language, vision, and rank-study evaluations. Across these experiments, settings vary by task while preserving controlled Adam–Muon comparisons.

  • D.4. A Fixed-Subspace LoRA Surrogate: The fixed-subspace LoRA surrogate freezes A, trains B from B0 = 0, and reduces fine-tuning to residual fitting with input u = Az.This constrains updates to a low-dimensional adapter geometry while retaining the residual-problem structure.
  • D.4. A Fixed-Subspace LoRA Surrogate: Proposition D.8 characterizes convergence and exact-fit adapter budgets for SignGD and idealized Muon under the fixed-subspace surrogate.The associated mismatch inflation factors quantify how adapter geometry changes the required budget.
  • D.4. A Fixed-Subspace LoRA Surrogate: In each native geometry, matched and mismatched exact-fit adapter budgets coincide, giving a mismatch inflation factor of one; A = I recovers full-fine-tuning thresholds.The appendix summary frames this as geometry-aligned budgeting for residual correction and old-task damage.
  • E. Experimental Details: The experiments use task-specific training schedules, validation procedures, batch sizes, sequence lengths, and learning-rate sweeps for controlled Adam–Muon comparisons.NLU uses T5-Base with 5 epochs for MRPC and CoLA, 3 for SST-2, QNLI, and MNLI, batch size 64, and sequence length 128.
  • E.1. Natural Language Understanding: For NLU, embeddings and the LM head use Adam while other parameters use Muon; LoRA experiments optimize their trainable parameters with Muon.Muon uses Nesterov momentum, Newton–Schulz iteration, and shape-dependent learning-rate scaling in the stated configurations.
  • E.2. Natural Language Generation: The NLG evaluation uses Llama 2-7B for one epoch and lm-evaluation-harness, while the 13B CodeFeedback extension averages HumanEval Pass@1 over 3 seeds.LoRA uses rank r = 8 and α = 16 for the 7B setup; the 13B comparison reports comparable LoRA-Muon and LoRA-Adam performance.
  • E.3. Image Classification: The image-classification experiments fine-tune CLIP ViT-B/32’s vision branch for 40 epochs with cached template-ensemble text features and task-specific optimizer assignments.The text tower is frozen, and matrix-shaped parameters use Muon while embedding-like and other non-matrix parameters use an alternative optimizer.
  • E.4. LoRA Rank Study: The LoRA rank study evaluates r ∈ {2, 4, 8, 16, 32, 64, 128, 256, 512} with α = 2r and learning-rate sweeps on MetaMath, CodeFeedback, and StanfordCars.MetaMath and CodeFeedback follow NLG settings, while StanfordCars follows image-classification settings.

E.5. Catastrophic Forgetting Evaluation · E.6. LoRA Variants · F. Additional Results

The catastrophic-forgetting evaluation measures retention of pretrained knowledge on unrelated commonsense benchmarks and quantifies weight movement relative to Adam. LoRA-variant experiments compare alternative methods and specify their benchmark outcomes and implementation settings.

  • E.5. Catastrophic Forgetting Evaluation: E.5. Catastrophic Forgetting Evaluation: The evaluation uses Llama 2-7B models fine-tuned on MetaMath and tests knowledge acquired during pretraining but unrelated to mathematics.Benchmarks showing improved performance after fine-tuning are excluded because they do not reflect forgetting.
  • F. Additional Results: F. Additional Results: The supplied passages contain no specific methods or findings for this section.No evidence passage labeled F. Additional Results was provided.
  • E.5. Catastrophic Forgetting Evaluation: E.5. Catastrophic Forgetting Evaluation: Commonsense evaluation covers ARC-Challenge, ARC-Easy, HellaSwag, OpenBookQA, and PIQA, while excluding WinoGrande and BoolQ.The exclusions avoid confounding forgetting with task-transfer effects.
  • E.5. Catastrophic Forgetting Evaluation: E.5. Catastrophic Forgetting Evaluation: Weight movement is measured using L2 and cosine distances between fine-tuned and pretrained weights, normalized so Adam = 1.0×.Values above 1 indicate greater movement than Adam, whereas values below 1 indicate smaller movement.
  • E.6. LoRA Variants: E.6. LoRA Variants: LoFT reaches 88.83% average on GLUE with T5-Base, below LoRA-Adam’s 88.93%, while LoRA-Pro and LoRA-RITE outperform LoRA-Adam.LoFT’s task scores are CoLA: 82.45±0.75%, MNLI: 86.14±0.07%, MRPC: 87.99±0.42%, QNLI: 93.15±0.11%, and SST-2: 94.42±0.24%.
  • E.6. LoRA Variants: E.6. LoRA Variants: AdaLoRA uses target average rank r = 8 to match the rank used in other methods.This setting is part of the variant-specific experimental configuration.
  • E.6. LoRA Variants: E.6. LoRA Variants: PiSSA uses full SVD for initialization, while other methods use defaults from their official implementations unless specified.The experiments follow each method’s default settings where no variant-specific change is reported.
  • E.6. LoRA Variants: E.6. LoRA Variants: LoRA-One uses stable gamma=64, torch.svd lowrank with q = 512 and niter=16, and batch size 1 for gradient estimation over 8 iterations.The approximate negative gradient is denoted −G.

F.1. Weight Spectral Analysis

During NanoChat pretraining, Muon produces attention QKV weights with consistently higher stable rank and SVD entropy than Adam, indicating a more distributed spectral structure across query, key, and value projections.

  • F.1. Weight Spectral Analysis: Stable rank measures a weight matrix’s effective dimensionality and whether its capacity is dominated by a few large singular values.A higher stable rank means the matrix utilizes more of its capacity.
  • F.1. Weight Spectral Analysis: SVD entropy quantifies singular-value dispersion on a normalized [0, 1] scale, with higher entropy indicating a more uniform distribution.The entropy measure captures how evenly singular values are distributed.
  • F.1. Weight Spectral Analysis: Muon-trained attention QKV weights consistently maintain higher stable rank and SVD entropy than Adam throughout NanoChat pretraining.Higher stable rank indicates greater effective dimensionality, while higher SVD entropy indicates a more uniform singular-value distribution.
  • F.1. Weight Spectral Analysis: The Muon–Adam difference is consistent across query (Q), key (K), and value (V) projections.Muon produces weights with higher stable rank and entropy for each projection type.

F.2. Spectral Analysis of LoRA Matrices · G. Computational Resources

Across Llama 2-7B fine-tuning, LoRA-Muon produces higher stable rank and normalized SVD entropy than Adam, extending Muon’s spectral bias to newly initialized LoRA matrices while preserving frozen pretrained weights. The experiments used AMD Instinct MI210, NVIDIA A6000, and NVIDIA A100 GPUs across T5, Llama, and CLIP models.

  • F.2. Spectral Analysis of LoRA Matrices: LoRA-Muon consistently yields higher stable rank than Adam across attention and dense layers, at ∼6–7 versus ∼3–5.The analysis covers Q/K/V/O attention projections and dense layers across MetaMath, CodeFeedback, and WizardLM.
  • F.2. Spectral Analysis of LoRA Matrices: LoRA-Muon also produces higher normalized SVD entropy than Adam, at ∼0.98–1.0 versus ∼0.80–0.95.These spectral differences are consistent across all analyzed layer types and tasks.
  • F.2. Spectral Analysis of LoRA Matrices: The spectral results indicate that Muon’s implicit bias toward uniform singular value distributions extends from pretrained weights to LoRA matrices.The reported stable-rank and entropy patterns mirror those observed in pretrained weights.
  • F.2. Spectral Analysis of LoRA Matrices: LoRA applies Muon to freshly initialized A and B matrices while frozen base weights preserve pretrained knowledge.This allows Muon to express its spectral bias on new matrices without directly modifying the pretrained parameters.
  • F.2. Spectral Analysis of LoRA Matrices: By contrast, full fine-tuning forces Muon to alter Adam-shaped pretrained weights directly, causing disruption.The contrast is offered as an explanation for why LoRA mitigates optimizer mismatch.
  • G. Computational Resources: T5-Base NLU and LoRA-variant experiments used a single AMD Instinct MI210 GPU.Training and evaluation both ran on that GPU.
  • G. Computational Resources: Llama 2-7B/13B experiments trained on 8× AMD Instinct MI210 GPUs and evaluated on 8× NVIDIA A6000 GPUs.These experiments included NLG, rank studies, and catastrophic-forgetting analyses.
  • G. Computational Resources: CLIP ViT-B/32 image-classification experiments trained and evaluated on a single NVIDIA A100 40GB GPU.The same single GPU was used for both phases.
Loading 2605.10468v1…