PaperlayerView exact revision on arXiv

Source-linked AI summary

The Hidden Power of Scaling Factor in LoRA Optimization

Zicheng Zhang, Haoran Li, Jiaxing Wang, Guoqiang Gong, Anqi Li, Yudong Hu, Ting Xiong, Yurong Gao, Junxing Hu, Zhida Jiang, Yifeng Zhang, Pengzhang Liu, Qixia Jiang

arXiv:2606.12883v1cs.AI

TL;DR

LoRA’s scaling factor α remains underexplored and is often tied to simplistic rank-based heuristics, despite its unclear optimization role. This paper combines empirical sweeps with a Signal-Drift framework to analyze α and η, finding that LoRA-α’s larger, sublinearly rank-scaled α consistently improves performance while enabling standard small learning rates.

  • Problem

    LoRA’s scaling factor α remains systematically underexplored and is typically tied to simplistic rank-based heuristics, leaving its optimization role unclear.

  • Method

    The paper combines systematic hyperparameter sweeps with a theoretical Signal-Drift framework to characterize how α, learning rate η, and rank affect LoRA optimization.

  • Results

    LoRA-α consistently improves performance across diverse models, tasks, and training paradigms while enabling direct reuse of standard small FFT learning rates.

  • Takeaways & Limitations

    Scaling α, rather than increasing η, provides a simple practical route to stronger LoRA fitting and reduced hyperparameter tuning.

  • Takeaways & Limitations

    The observed asymmetry between α and η is grounded in adaptive optimizers and may not directly extend to vanilla SGD.

Abstract

from arXiv · show

In Low-Rank Adaptation (LoRA), the scaling factor $α$ is often treated as a mere complement to the learning rate, yet its role in optimization remains poorly understood. In this paper, we reveal that the scaling factor $α$ and the learning rate function differently, with $α$ emerging as the dominant driver of effective optimization, delivering gains that cannot be replicated by learning rate scaling alone. Through the synergy of extensive empirical analysis and a theoretical Signal-Drift framework, we uncover three findings into LoRA's scaling mechanism: First, LoRA's spectral suppression smooths the optimization landscape, rendering standard hyperparameters overly conservative and creating an optimization gap. Second, when leveraging this smoothness to accelerate convergence, $α$ outperforms the learning rate by amplifying the task signal without increasing the drift ratio. Third, the optimal scaling factor follows a sublinear relationship with the rank, well characterized by a square-root law with an unexpectedly large coefficient, revealing the insufficient scaling of existing rank-tied heuristics. Based on these insights, we propose LoRA-$α$, a minimalist framework that restores $α$ to its principled regime, making LoRA compatible with standard small learning rates. Extensive evaluations across diverse tasks demonstrate that LoRA-$α$ consistently improves performance while streamlining hyperparameter search, unleashing the learning potential of LoRA.

1 Introduction

The introduction identifies LoRA’s scaling factor α as an underexplored determinant of optimization, more important than learning-rate increases, and presents a Signal-Drift analysis of its role. It proposes LoRA-α, using large square-root scaling to improve optimization while enabling standard FFT learning rates and reducing tuning.

  • Background: LoRA represents weight updates as ΔW = α/r BA, enabling parameter-efficient adaptation across NLP and multimodal generation.Low-rank factors B and A approximate the update with rank r and scaling factor α.
  • Motivation: Despite LoRA’s adoption, α remains underexplored and is commonly tied to simplistic rank heuristics such as α = r or 2r.These practices obscure α’s role in reducing hyperparameter search and shaping the optimization regime.
  • Key findings: Extensive sweeps show that effective optimization depends more on sufficiently large α than on increasing the learning rate η.The paper studies this behavior through a joint empirical and theoretical Signal-Drift framework.
  • Key findings: LoRA’s low-rank parameterization spectrally suppresses the task Hessian, smoothing the landscape but making standard hyperparameters overly conservative and creating an optimization gap.This smoothing motivates the aggressive scaling practices observed in prior work.
  • Proposed method: LoRA-α restores α to a principled regime with a large base value and square-root scaling, allowing standard FFT learning rates without costly hyperparameter tuning.The proposal addresses insufficient magnitudes in prevailing rank-tied heuristics and aims to recover underutilized fitting potential.

2 An Empirical Study of LoRA Scaling

Systematic fitting-focused sweeps show that increasing LoRA’s scaling factor α improves optimal loss while reducing the optimal learning rate, and that α-scaling reaches deeper minima than learning-rate scaling. Across four optimization settings, the optimal α follows a sublinear rank relationship but remains substantially larger than conventional choices.

  • Interplay between α and η: Increasing α consistently lowers the optimal loss while decreasing the optimal learning rate η∗, reversing LoRA’s need for roughly 10× larger learning rates than FFT at α = 16.The analysis isolates fitting dynamics from generalization effects and evaluates Expected Negative Log-Likelihood on supervised fine-tuning configurations.
  • Superiority of α-Scaling over η-Scaling: α-scaling supports a substantially larger optimal multiplier and converges to deeper loss minima, whereas η-scaling prematurely plateaus.Both paths begin from ηFFT = 2 × 10−5 and α0 = 16 and exhibit U-shaped loss surfaces.
  • Scaling Law between α and r: The optimal scaling factor α∗ follows a sublinear relationship with rank r, rather than the linear rank dependence used by conventional scaling.The relationship is defined by α∗(r) = arg minα L(r, ηFFT, α).
  • Scaling Law between α and r: α ≥256 across all four optimization settings, indicating that conventional scaling is severely under-scaled under typical FFT learning rates.The paper attributes the fitting limitation to both conventional scaling’s linear dependence on r and its insufficient magnitude.
  • Relationship to Prior Works: The results attribute LoRA’s reported requirement for 10× larger learning rates than FFT to insufficient α scaling and provide a concrete rank-dependent scaling law.Prior work reported the larger learning-rate requirement, while earlier scaling studies either used α = r, explored only up to α = 2r, or identified mild rank dependence without its appropriate magnitude.

3 A Theoretical Analysis from the Signal-Drift Perspective

The Signal-Drift framework decomposes LoRA updates into task-aligned signal and bilinear-parameterization drift, showing that signal promotes descent while drift introduces task-agnostic instability. It explains LoRA’s optimization behavior through spectral suppression, asymmetric α/η scaling, and a sublinear rank-scaling law for α.

  • Signal-Drift framework: The Signal-Drift decomposition separates LoRA’s effective update and parameter-space Hessian into task-aligned signal and structural drift.The decomposition is exact under LoRA’s bilinear parameterization.
  • Geometric properties: Signal aligns with descent and preserves local convexity, whereas strictly indefinite drift has no guaranteed alignment with the task gradient.The signal satisfies ⟨∆wSignal, −g⟩≥0 and HSignal ⪰0 when Hℓ⪰0; drift introduces uncontrolled, task-agnostic instability.
  • Spectral suppression: ρ ≪1 under standard initialization compresses LoRA’s signal curvature and expands its stable learning-rate bound, making FFT hyperparameters overly conservative.The bound is η ≤2/λmax[HSignal].
  • Asymmetric scaling: Increasing η degrades the Signal-to-Drift Ratio as Θ(1/η), while α increases signal relative to drift and reshapes the landscape toward the signal spectrum.Under Adam-like optimizers, ∆wSignal = Θ(αη) and ∆wDrift = Θ(αη^2).
  • Rank-dependent scaling: The optimal α should scale sublinearly with rank and grow substantially with model width, because conventional linear scaling fails to restore the task’s curvature scale.This follows from matching expected LoRA signal curvature to full-task curvature at initialization.

4 The LoRA-α Protocol

LoRA-α calibrates the scaling factor to restore LoRA’s suppressed curvature, enabling the standard FFT learning rate while improving fitting capacity over learning-rate scaling alone. It separates curvature restoration from lightweight tuning and offers empirical and analytic strategies for determining the base scale.

  • Protocol principle: Calibrating LoRA’s signal curvature to FFT enables direct use of the standard FFT learning rate without costly hyperparameter sweeps.This calibration also unlocks stronger fitting capacity than learning-rate scaling alone.
  • Protocol principle: LoRA-α sets αbase to restore LoRA’s suppressed curvature, while α can default to 1 or be searched narrowly within [0.1, 10].This decouples scaling from optimization and avoids expensive joint tuning over scaling factors and learning rates.
  • Scaling strategies: For modern LLMs, empirical scaling uses C = 256, yielding αbase = 256√r across architecturally homogeneous models.The method targets models with hidden dimensions on the order of thousands, such as d_in = 4096.
  • Scaling strategies: Analytic scaling derives αbase by matching expected task curvature at initialization, providing a theoretical baseline aligned with FFT’s hyperparameter regime.This strategy is intended for general architectures and supplies layer-wise theoretical initialization.
  • Scaling strategies: Variant I produces marginally larger magnitudes than Variant II for modern LLMs, a gap attributed plausibly to trace-of-Hessian averaging in analytic scaling.Variant I is a global empirical heuristic, whereas Variant II is a layer-wise theoretical initialization.

5 Empirical Validation of LoRA-α

Across diverse models, tasks, and training horizons, LoRA-α consistently matches or outperforms existing LoRA variants and learning-rate scaling. Its advantages extend from NLU and multimodal generation to long-horizon retrieval, reasoning, and reinforcement learning.

  • Short-horizon adaptation: LoRA-α achieves the strongest or near-strongest average performance across eight GLUE NLU datasets, outperforming vanilla LoRA and alternative scaling strategies.DeBERTa-v3-base-184M models use batch size 32, learning rate 1 × 10^-4, rank r = 8, and 3–5 epochs.
  • Short-horizon adaptation: Across mathematical reasoning, code generation, and commonsense NLG tasks, LoRA-α matches or surpasses existing LoRA variants across ranks r ∈ {16, 128}.Llama 2-7B models train for one epoch over 100k samples with batch size 128 and learning rate 2 × 10^-5.
  • Short-horizon adaptation: LoRA-α produces higher-fidelity Flux.1-12B images than PiSSA and LoRAM, showing that improved scaling transfers to multimodal generative settings.DreamBooth training uses rank r = 8, learning rate 1 × 10^-4, batch size 1, and 1,000 flow-matching steps.
  • Scaling comparison: α-scaling consistently outperforms η-scaling on NLU and NLG tasks, with the performance gap widening significantly at larger ranks.The comparison is based on hyperparameter sweeps reported in Tables 1 and 2.
  • Long-horizon adaptation: On MMEB retrieval, LoRA-α consistently outperforms PEFT baselines and FFT across 2B and 7B models, with the largest gains on OOD evaluation.The method also surpasses the best official VLM2Vec checkpoint without architectural modifications.
  • Reasoning and reinforcement learning: In long-context reasoning and GRPO, LoRA-α gains strength at higher ranks, remains closest to FFT, and achieves the best overall performance across both reinforcement-learning model scales.Learning-rate scaling becomes unstable at higher ranks in reasoning and often degrades GRPO performance, while FFT is inconsistent or remains superior on difficult reasoning tasks.

6 Conclusion

The paper explains LoRA’s scaling mechanisms through empirical evidence and a Signal-Drift framework, showing that spectral suppression creates an optimization gap and that scaling α is more effective than increasing the learning rate. It also establishes a sublinear scaling law and identifies drift-mitigating adapter architectures as a future research direction.

  • Key findings: LoRA’s stability stems from spectral suppression of the task Hessian, which creates a significant optimization gap.The conclusion attributes this finding to the paper’s combined empirical evidence and Signal-Drift framework.
  • Key findings: Scaling α is more effective than increasing the learning rate because it amplifies task-aligned signal without introducing additional drift.This conclusion directly contrasts the two scaling mechanisms within the Signal-Drift framework.
  • Key findings: The paper establishes a sublinear scaling law for LoRA’s scaling mechanism.The supplied conclusion states this result without specifying the law’s exact functional form.
  • Future work: Future research should develop adapter architectures that structurally mitigate drift without compromising task-signal integrity.The passage presents this as an exciting frontier for future work.

A Limitations … D.2 Scaling Behavior of Llama 3-8B on Tulu 3

The paper frames LoRA-α as a scaling strategy whose benefits extend across optimization analysis, empirical methodology, and large-model evaluation, while noting limits to its optimizer-specific interpretation. On Llama 3-8B, α-scaling follows a square-root law and enables standard-small-learning-rate training to approximate FFT dynamics.

  • A Limitations: The asymmetric roles of α and η may not directly extend to vanilla SGD, although the Signal–Drift decomposition itself is optimizer-agnostic.The analysis reflects practical training regimes dominated by adaptive optimizers.
  • B Broader Impact: The proposed scaling strategy may reduce adaptation costs and broaden access to large-model fine-tuning, while improved deployment could also facilitate misuse.These are potential broader effects rather than experimentally established outcomes in the supplied passage.
  • C Related Work: LoRA enables efficient large-model adaptation through low-rank decomposition, reducing training and serving costs in scalable modern systems.The passage identifies LoRA as a key technique and points to surveys covering its extensions.
  • C Related Work: Prior analyses often simplify LoRA’s bilinear, non-convex optimization, while stability studies show improper scaling can cause gradient collapse or inefficient training.The supplied passage contrasts simplified theoretical regimes with stability-focused work.
  • C Related Work: LoRA is commonly trained with α ∝r while tuning η, despite often requiring learning rates exceeding 10× those used for full fine-tuning.Recent theory also reports coupling between η and α, with increasing α able to mimic part of the learning-rate effect.
  • C Related Work: The study attributes LoRA’s aggressive hyperparameter requirements to a key asymmetry: α amplifies task-aligned curvature, whereas η amplifies bilinear-induced drift.This discussion positions the work as the first systematic characterization of α’s role in LoRA optimization, according to the passage.
  • D.1 Experimental Setup and Search Strategy: For Tulu 3, the search used a coarse η–α grid followed by localized refinement around the minimum validation loss to identify η∗ and α∗.The broader empirical section covers experiments across model scales and datasets.
  • D.1 Experimental Setup and Search Strategy: For OpenThoughts, the search centered on C = 256 and swept α across a 10× range to test transferability of the sublinear scaling law.The dataset used a 10k training subset, a 1k validation set, and maximum sequence length T = 8192.

D.3 Validation on OpenThoughts Reasoning Tasks

On OpenThoughts reasoning tasks, scaling α preserves its advantage over learning-rate tuning across Llama 3-1B and 8B, enabling deeper loss minima without early saturation. The optimal scaling follows the established sublinear rank law, with fitted coefficients C = 1024 indicating especially aggressive scaling demands.

  • Optimization behavior: Both models avoid early saturation when α is scaled, while learning-rate scaling does not provide the same optimization behavior.The persistent α–η asymmetry is observed on both OpenThoughts model sizes.
  • Scaling law: C = 1024 indicates that complex reasoning tasks may require especially aggressive restoration of α.The large fitted coefficient is reported for the OpenThoughts optimization behavior.
  • Optimization behavior: Scaling α lets both Llama 3-1B and 8B reach deeper loss minima that η-tuning alone cannot access.This asymmetry persists despite OpenThoughts’ longer sequences and greater task complexity.
  • Scaling law: The fitted optimal scaling relationship remains sublinear in rank, confirming the main-text scaling law across OpenThoughts tasks.Figures 6 and 7 show α* as a function of rank r for Llama 3-1B and 8B.

E Empirical Validation of Signal-Drift Dynamics

Controlled MLP experiments validate Signal-Drift dynamics: low-rank parameterization suppresses curvature, while increasing α strengthens task-aligned signal more safely than learning-rate scaling. This explains why α-scaling enables stable, accelerated optimization where aggressive learning-rate scaling can diverge.

  • Experimental setup: Experiments use a 5-layer SiLU MLP fitting synthetic Gaussian data, with rank r = 16 LoRA adapters and exact Hessian analysis at d = 32.General optimization tracking uses d = 256; the target singular values follow σ_i = 1/i.
  • Optimization dynamics: η = 128η0 catastrophically diverges, whereas α = 256α0 remains strictly stable and continues deeper fitting after comparable initial convergence at η = 64η0 and α = 128α0.The η-scaled trajectory already shows noticeable oscillations at η = 64η0.
  • Spectral compression: The LoRA signal Hessian spectrum is compressed relative to the full-parameter Hessian, suppressing curvature energy and requiring larger α to compensate for rank-induced suppression.The maximum eigenvalue follows the predicted order of magnitude, while trace-level compression necessitates suppression of dominant spectral components.
  • Spectral purification: Structural drift makes HLoRA indefinite by introducing negative eigenvalues, but increasing α amplifies signal relative to drift and progressively purifies the spectrum toward HSignal.The spectral purification effect aligns the indefinite spectrum increasingly with the positive semi-definite signal Hessian.
  • Asymmetric scaling: Because α disproportionately strengthens task-aligned curvature while only linearly amplifying drift, the dominant HLoRA eigenvalue increasingly aligns with HSignal.The experiments identify α, rather than the learning rate, as the principled route to stable and accelerated optimization.

F Comparison of Scaling Factor Magnitudes · G Theoretical Proofs · G.1 Proof for Proposition 1

The paper argues that conventional α = r scaling is severely under-scaled, while LoRA-α uses substantially larger magnitudes guided by empirical and analytic scaling choices. It then proves that LoRA updates and Hessians decompose exactly into task-aligned signal and structural drift.

  • F Comparison of Scaling Factor Magnitudes: Conventional α = r scaling confines LoRA to a severely under-scaled regime as rank increases, motivating larger scaling magnitudes.The Signal-Drift framework links restoring curvature energy comparable to FFT with scaling α according to the proposed law.
  • F Comparison of Scaling Factor Magnitudes: For rank r = 8, LoRA-α uses a scaling magnitude 90.5× larger than the common heuristic.At r = 256, the gap remains 16×.
  • G.1 Proof for Proposition 1: The Signal-Drift proposition states that LoRA’s effective weight step and parameter-space Hessian each admit exact decompositions into task-aligned signal and structural drift.These decompositions are formulated under the LoRA parameterization for the effective step Δw_LoRA and Hessian H_LoRA.
  • G.1 Proof for Proposition 1: Because LoRA’s parameter-to-weight mapping is strictly bilinear, all third-order and higher-order mapping derivatives vanish identically.Consequently, the Taylor expansion has no O(∥Δθ∥^3) residual and terminates exactly at second order.
  • G.1 Proof for Proposition 1: The Hessian decomposition follows by applying the multivariate chain rule and product rule, separating the J(θ)^T H_ℓ J(θ) term from the remaining structural contribution.The proof converts the element-wise equality into matrix notation and establishes both partitions.
  • G.1 Proof for Proposition 1: For perturbations Δθ = {ΔA, ΔB}, the first-order terms in ΔA and ΔB form the task signal, while their bilinear cross-term forms structural drift.This follows directly from expanding the LoRA forward pass and vectorizing the linear and bilinear components.

G.2 Proof for Proposition 2 · G.3 Proof for Proposition 3

The proofs establish that LoRA’s signal aligns with descent and preserves local convexity, whereas bilinear drift creates indefinite curvature and can oppose descent. They also derive the expected signal-curvature scaling and show that spectral suppression expands the stable learning-rate range.

  • G.2 Proof for Proposition 2: The signal component aligns with the descent direction, satisfying ⟨∆wSignal, −g⟩≥0.This alignment is established across GD, Adam’s sign-gradient abstraction, and Muon’s orthogonalized update.
  • G.2 Proof for Proposition 2: When Hℓ⪰0, the signal Hessian HSignal = J(θ)⊤HℓJ(θ) is positive semi-definite.The proof evaluates v⊤HSignalv through u = J(θ)v and uses u⊤Hℓu ≥0.
  • G.2 Proof for Proposition 2: Because HDrift is indefinite, the drift update has no guaranteed nonnegative alignment with −g and can oppose the descent direction.The drift-direction inner product reduces to a quadratic form that is not guaranteed to be non-negative.
  • G.2 Proof for Proposition 2: Bilinear LoRA parameterization makes HDrift block off-diagonal with eigenvalues in symmetric ±λ pairs, establishing an indefinite saddle structure.This disruptive geometry is independent of the task curvature Hℓ.
  • G.3 Proof for Proposition 3: The expected signal curvature satisfies E[Tr(HSignal)] = α2ρ Tr(Hℓ), linking curvature directly to the scaling factor and initialization variance term ρ.The derivation uses LoRA’s vectorized mapping, initialization with B = 0, and Gaussian initialization of A.
  • G.3 Proof for Proposition 3: At initialization, B = 0 causes the Jacobian block JA to vanish completely, while the remaining Jacobian block corresponds to B.The signal curvature is projected as HSignal = J⊤HℓJ.
  • G.3 Proof for Proposition 3: Reducing λmax(H) through spectral suppression increases the upper bound of the stable learning rate and expands the permissible hyperparameter space.For gradient descent, stability is governed by λmax(H), with the bound η ≤2/λmax(H).

G.4 Proof for Proposition 4 … H.2 Natural Language Generation

The paper proves that α and η affect LoRA’s landscape and adaptive-optimizer updates asymmetrically, then specifies evaluation configurations for NLU and NLG experiments. These evaluations use fixed optimizer settings, task-specific datasets and metrics, and analytically determined scaling for LoRA-α.

  • G.4 Proof for Proposition 4: α scales the signal Hessian quadratically but the drift Hessian linearly: HSignal = Θ(α2) and HDrift = Θ(α).The Jacobian and structural tensor are proportional to α, producing quadratic signal scaling through an outer product and linear drift scaling.
  • G.4 Proof for Proposition 4: Under Adam-like adaptive optimization, the update signal scales as Θ(αη), whereas update drift scales as Θ(αη2).Adaptive normalization makes the parameter-update magnitude Θ(η), after which signal is first-order in the update and drift is quadratic in η.
  • G.4 Proof for Proposition 4: The proof derives these relationships by establishing α-scaling for mapping derivatives and propagating it through landscape and update decompositions.Because the LoRA mapping is bilinear in A and B, its nonzero second derivatives are cross-derivatives that retain linear α dependence.
  • H Experimental Details: Table 7 summarizes configurations and hyperparameters across six evaluation domains, while later subsections provide dataset, metric, and task-specific details.Learning-rate schedules and environmental configurations follow the referenced official codebases.
  • H.1 Natural Language Understanding: NLU experiments use DeBERTa-v3-base (184M) on eight GLUE tasks, with rank r = 8, adapters in all linear layers, and LoRA-α’s analytic Variant II scaling.Training uses AdamW with a constant learning rate of 1 × 10−4 for three epochs on most tasks, five for MRPC; results average three runs with GLUE metrics.
  • H.2 Natural Language Generation: NLG experiments fine-tune Llama 2-7B across mathematical reasoning, code generation, and commonsense reasoning domains.Datasets include MetaMathQA, CodeFeedback, and Commonsense170K, evaluated respectively on GSM8K/MATH, HumanEval/MBPP, and the associated commonsense benchmarks.
  • H.2 Natural Language Generation: NLG compares rank regimes r ∈{16, 128} using LoRA-α’s analytic Variant II scaling, with adapters applied uniformly to all linear layers.Training uses AdamW, learning rate 2 × 10−5, total batch size 128, one epoch, and three-run averages; Table 8 reports standard deviations.

H.3 Detailed Hyperparameter Search Results … H.7 Experimental Details for Reasoning-based RL

Across hyperparameter searches and experiments spanning generation, multimodal retrieval, supervised reasoning, and reinforcement learning, LoRA-α uses structural scaling to improve optimization and is evaluated under detailed task-specific protocols. The results show higher aggregated performance and greater stability than η-scaling, alongside configurations for diverse models, datasets, and reasoning benchmarks.

  • H.3 Detailed Hyperparameter Search Results: 89.03 aggregated peak performance for LoRA-α⋆ exceeds 88.53 for η-scaling in the NLU grid search.η-scaling often reaches suboptimal plateaus or local instability, including initial failure to converge on CoLA.
  • H.3 Detailed Hyperparameter Search Results: 44.87 average performance for α-scaling exceeds 42.69 for η-scaling in the NLG grid search, while preserving stability at larger ranks.At r = 128, extreme η-scaling can collapse performance to 0.00 on MATH and GSM8K, whereas α-scaling reaches deeper local minima.
  • H.4 Text-to-Image Synthesis: Text-to-image synthesis evaluates Flux.1-12B with DreamBooth on typically 4–6 instance images, using LoRA-α analytic Variant II scaling with r = 8.Adapters are applied to all attention layers, and evaluation tests concept preservation in category and scene contexts.
  • H.5 Multimodal Representation Learning: Multimodal representation learning evaluates LoRA-α on MMEB with Qwen 2-VL 2B and 7B backbones, using Variant I with αbase = 256√r and r = 16.Training uses 20 in-distribution subsets, while 16 out-of-distribution subsets are reserved for zero-shot evaluation; performance is measured by Precision@1.
  • H.6 Experimental Details for Reasoning-based SFT: Reasoning SFT uses Qwen 2.5-Math-7B on the 350k-sample Mixture-of-Thoughts dataset, with adapters at ranks r ∈{64, 256} and Variant I scaling.Embedding and output layers are additionally trained, with evaluation using Pass@1 accuracy across AIME 24/25, MATH-500, GPQA Diamond, and LiveCodeBench v4.
  • H.7 Experimental Details for Reasoning-based RL: Reasoning RL uses GRPO with DeepSeek-R1-Distill-Qwen 1.5B and 7B models on DAPO-Math-17k, implementing an RLVR pipeline based on PeRL.Adapters use rank r = 64 and Variant I; training generates G = 8 rollouts per prompt and evaluates Pass@1 accuracy over multiple samples.
Loading 2606.12883v1…