Scale-QLoRA: Code-Invariant Adapter Merging for Native 4-bit Microscaling LLMs

1 INTRODUCTION

• A merged checkpoint faces re-export, format conversion, re-quantization, deduplication, hot-swap, and rollback events, while E2M1 codes account for roughly 90% of its bytes.Format or grid conversion re-quantizes both lanes and is outside the stated guarantee.Source: §1 · p. 1, Figure 1 · p. 2
• The naive dequantize-add-requantize merge costs 37.4 pp on NVFP4 Llama-8B Banking77 (93.0 →55.6), with losses up to 39 pp across four post-hoc quantizers and tasks.On an already-on-grid base, the reconstruction optimum is the base model, so the deletion requires no quantizer mismatch.Source: §1 · pp. 1–2
• Both merge-aware methods are accuracy-lossless, but QAT-LoRA re-derives E2M1 codes while Scale-QLoRA freezes them; the paper claims no accuracy ordering between the methods.A nearest-rounding mismatch costs about a point on the measured task; the ∼0% result under an outside-family rule is a sensitivity bound, not a measured deployment frequency.Source: §1 · p. 2
• Scale-QLoRA learns a low-rank correction to per-block scales on the native grid, freezes all codes, and merges by writing the trained scale bytes and copying codes verbatim.Its bit-exact guarantee is within a fixed native format, scale grid, block layout, and code plane.Source: §1 · p. 2
• Sharing a code plane cuts storage 3.0× at M=4, and scale-only swapping is ∼125× faster than a weight-space swap that re-quantizes the checkpoint.The paper claims no latency advantage over an all-resident pointer swap or a runtime-adapter system.Source: §1 · p. 2
• Scale-QLoRA is a low-rank parameterization of the representational middle between per-row and weight-space adaptation, a distinction that does not establish accuracy equivalence.The paper identifies code-invariance and lifecycle analysis as its contribution beyond neighboring scale-adaptation methods.Source: §1 · p. 2

2 BACKGROUND AND THE MERGE VULNERABILITY

• A native FP4 weight is a 4-bit E2M1 code times a shared block scale; NVFP4 stores scales as E4M3 bytes times a per-tensor scalar, while MXFP4 uses E8M0 exponent bytes.The formats use blocks of 16 and 32 respectively, and both are symmetric with no zero-point.Source: §2 · p. 3
• The naive merge requantizes dequantized weights plus the LoRA delta, producing new codes and scales through a step the trained model never saw.The stated design condition for an exact merge is that the field written at merge already matches what the model trained on.Source: §2 · p. 3
• Scale-only PEQA and AlphaTuning freeze codes and move scales, whereas QLoRA, GPTQ, AWQ, LoftQ, and related methods still merge through weight-space re-quantization.PEQA adapts one scale per output channel; weight-space QLoRA adapts a higher-capacity low-rank delta off the native grid.Source: §2 · p. 3
• Native-grid STE training makes QAT-LoRA’s merge reproduce training, but its guarantee depends on the deployment tool applying the exact training rule to every code.The method also re-quantizes weights at every forward step, increasing training cost.Source: §2 · p. 3
• Scale-QLoRA is the zero-point-free microscaling analogue to QA-LoRA, using per-block scales as the absorbing parameter and QAST to target the hardware grid.Unlike LoRDS, it adapts existing per-block scale bytes into a standard-format checkpoint rather than relying on continuous scaling served through custom kernels.Source: §2 · p. 3

3 METHOD: SCALE-QLORA

• The method’s guarantee is limited to a fixed native format, scale grid, block layout, and code plane; format conversion re-quantizes both lanes.Bit-exactness and code-invariance follow from construction, while accuracy-losslessness is an empirical downstream property.Source: §3 · pp. 3–5
• For the dense model, Scale-QLoRA is 3.9× cheaper per training step than QAT-LoRA; the passage makes no accuracy ordering between the two merge-aware paths.Source: Figure 2 · p. 4
• Scale-QLoRA learns a low-rank correction to per-block scales, not weights, while leaving the codes unchanged.Its block-level factor A has one column per scale block, so it carries fewer trainable parameters than weight-space QLoRA at equal rank.Source: §3 · p. 4
• QAST rounds the effective scale onto the native grid during each forward pass, so merge writes the scale value already used in training and copies every code unchanged.The scale is quantized during training; the guarantee is that merge applies no second, unseen quantization.Source: §3 · pp. 4–5, Figure 2 · p. 4
• Scale-QLoRA achieves max |∆W| = 0 at merge on both native grids, including E8M0 MoEs up to 100B+ parameters, when the merge grid matches the training grid.A mismatched QAST-e4m3 adapter merged onto E8M0 loses −13.0 pp on Banking77 and −8.2 pp on AGNews.Source: §3 · p. 5
• The single-setting recipe combines validation early stopping, an E4M3 ceiling clamp, and a merge path matching the forward grid-and-clamp path.Without the clamp, Qwen-30B first becomes non-finite at step 18; omitting the clamp at merge yields NaNs and an artificial ∼−92 pp merge loss.Source: §3 · p. 5

4 EXPERIMENTAL SETUP

• The evaluation spans four native-microscaling checkpoints: Llama-8B and Qwen-30B on NVFP4/E4M3, plus gpt-oss-120B and DeepSeek-V4-Flash on MXFP4/E8M0.The two 100B+ MoEs use the coarse power-of-two E8M0 scale grid.Source: §4 · p. 5
• The main task matrix covers Banking77, AGNews, CLINC150, and Spider; MBPP is an additional generative evaluation rather than a main-matrix cell.Spider uses execution match, while MBPP reports pass@1 against problem asserts in a sandbox.Source: §4 · p. 6
• Scale-QLoRA, QAT-LoRA, and RTN-merge are compared with two unquantized references; the four RTN-merge quantizers reuse one naive adapter without per-quantizer retraining.GPTQ is also used as calibrated PTQ; fp16_lora+PTQ is named as a workflow but is not a measured row.Source: §4 · p. 6
• Every task uses full training data, up to three epochs, validation early stopping, a uniform 5 × 10−5 learning rate, and parameter-matched scale ranks.The merge-loss comparison uses merged versus unmerged models on the native grid, with exactness separately checked by max |∆W|.Source: §4 · p. 6
• Main-matrix confidence intervals are roughly ±0.5–1 pp, but most rows are single-seed and Spider has run-to-run variation of 1.84 pp.The authors treat gaps below the stated CI, or about 2 pp on Spider, as indistinguishable from noise and claim no within-floor ordering.Source: §4 · p. 6
• Deployment experiments use vLLM 0.23.0 with modelopt-NVFP4 on H200, and merged and unmerged values for a cell are measured on the same backend.The 120B and DeepSeek models use bridge or log-probability evaluation rather than in-EP generation.Source: §4 · p. 6

5.1 Both merge-aware methods are accuracy-lossless; only one is bit-exact

• No accuracy ordering between Scale-QLoRA and QAT-LoRA is supported; repeated Spider runs are indistinguishable, and classification gaps remain unresolved.Seven Llama Spider runs give Scale-QLoRA 69.88 ± 0.83 versus QAT-LoRA 69.38 ± 1.15, with a 95% CI for the difference of [−0.55, +1.55].Source: §5.1 · p. 7
• Both merge-aware methods have |∆| ≤0.2 pp merge loss on every cell, but only Scale-QLoRA is bit-exact; RTN-merge is lossy on every cell but one.Scale-QLoRA has max |∆W| = 0 exactly, while QAT-LoRA has max |∆W| ∼3 × 10−3; the table reports no accuracy ordering between the merge-aware methods.Source: §5.1 · p. 7, Table 1 · p. 8

5.2 Code-invariance and its five payoffs

• 91.6–97.6 pp is QAT-LoRA’s per-dataset best-to-worst spread across five deployment rules, while Scale-QLoRA’s spread is 0.0 pp on every dataset.The QAT-LoRA rows are five re-quantizations of one merged tensor; Scale-QLoRA’s stored artifact is read across the columns without applying a rule.Source: §5.2 · p. 8, Figure 3 · p. 9
• About one point of cross-tool divergence is measured for nearest rounding, and later code-touching events require an exactly matching quantizer for QAT-LoRA artifacts.Only round-to-nearest is substantiated as shipping behaviour; the larger mismatches are sensitivity bounds, not measured deployment frequency.Source: §5.2 · p. 9, Figure 3 · p. 9
• 224/224 Llama-8B target-linear code planes remain identical across four Scale-QLoRA task adapters, enabling 3.0× less storage at four tasks and byte-verified rollback without a second base checkpoint.Scale-only task swaps are ∼125× faster than a weight-space swap that re-quantizes the checkpoint; this comparison does not establish an advantage over pointer swaps.Source: §5.2 · p. 9

5.3 Why the naive merge collapses: the reconstruction objective returns the un-adapted base

• On gpt-oss-120B AGNews, MSE costs −16.8 pp versus RTN’s −2.6 pp; the un-adapted block is the global reconstruction optimum on 99.4% of 140,000 measured blocks.The MSE merge lands at base accuracy (75.6 vs base 74.0), while RTN’s fixed rule bumps block exponents and retains essentially all the gain.Source: §5.3 · p. 10
• GPTQ recovers gpt-oss AGNews from 75.6 to 92.1, while an AWQ exception on DeepSeek AGNews breaks the reconstruction-quality ordering.The NVFP4 weight-displacement and accuracy instruments disagree, so the paper leaves that discrepancy open rather than resolving it in favor of either measure.Source: §5.3 · p. 11

5.4 Why adapting only the scales suffices

• Independent scale adaptation is not distinguishable from weight-space LoRA at the tested protocols, but the result is a null, not evidence of equivalence.The full-precision control is single-seed on one dense model and two classification tasks; the representational families remain nested as per-row ⊊ per-block frozen-code ⊊ weight-space.Source: §5.4 · pp. 11–12

6 SCOPE AND DISCUSSION

• Both merge-aware methods are accuracy-lossless (≤0.2 pp against full-data CIs of ±0.5–1 pp), so the paper claims no accuracy ordering between them.Scale-QLoRA preserves the code plane by construction; QAT-LoRA re-derives it through a quantizer.Source: §6 · p. 12
• Within a fixed native format, scale grid, block layout, and code plane, Scale-QLoRA merging is a bit-exact identity; format conversion lies outside that guarantee.QAST targets a specific hardware scale grid, so train-time grid selection must match deployment; a QAST-e4m3 adapter merged to E8M0 loses 13.0/8.2 pp.Source: §6 · p. 12
• The one measured lifecycle event separating the methods is in-family export-tool divergence, about a point on the measured task.Code-plane payoffs are measured on one dense model with a partial MoE replication, and only against merged-artifact designs.Source: §6 · p. 12
• A weight-space artifact can reach ∼0% under an out-of-family rule mismatch, a sensitivity bound rather than a measured deployment frequency.The authors also leave unresolved the weight-probe-versus-accuracy disagreement on NVFP4 rounding cells and the zero-delta base control’s non-zero reading on the finer grid.Source: §6 · p. 12
• The study spans a dense 8B model and three MoEs across four main-matrix tasks, but every row beyond specified classification cells is single-seed.Spider is the noisiest metric (1.84 pp run-to-run at fixed seed); a Scale-QLoRA result of 70.2 did not reproduce, returning 51.8.Source: §6 · p. 12
• The deploy sweep is narrower than it appears: it is confounded in one column, asymmetric by design, and prices no lifecycle event.GPTQ error feedback also needs a per-expert Hessian that sparse MoE calibration cannot supply; this limits calibrated PTQ guarantees, not these rows.Source: §6 · p. 12

7 CONCLUSION

• Scale-QLoRA and QAT-LoRA are both accuracy-lossless, with no general accuracy ordering; their distinction is that Scale-QLoRA preserves codes while QAT-LoRA re-derives them.The Scale-QLoRA merge is a bit-exact identity within a fixed native format, scale grid, block layout, and code plane.Source: §7 · pp. 12–13
• Scale-QLoRA freezes E2M1 codes and trains a low-rank correction to block scales on the native grid, making its merged artifact code-invariant.QAT-LoRA’s weight-space artifact is sensitive to quantization conventions; an in-family tool costs about a point on the measured task, while an out-of-family rule drives it to ∼0% as a sensitivity bound.Source: §7 · pp. 12–13
• A shared code plane enables bitwise-identical codes across four task adapters, 3.0× storage sharing at M=4, and scale-only task swaps ∼125× faster in the one-resident-checkpoint experiment.The conclusion also reports byte-for-byte rollback without a second base checkpoint and distribution patches 21.1× smaller than a weight-space merge.Source: §7 · p. 13
• Training is 3.9× cheaper per step on the measured dense 8B model because Scale-QLoRA removes the weight-space STE.The authors claim only zero per-forward overhead against runtime-adapter serving, not a latency advantage.Source: §7 · p. 13
• The reported lifecycle payoffs compare merged artifacts; against a retained base plus runtime adapters, the paper claims only zero per-forward overhead.Format conversion re-quantizes both methods’ artifacts, and the finer-grid zero-delta control leaves a non-zero reading that remains open.Source: §7 · p. 13
• Future work includes end-to-end E8M0 training at scale, broader generative tasks, full-data expressivity studies, and direct comparison with continuous low-rank scaling.The paper states its contribution is target-grid-aware low-rank adaptation of native microscale bytes, not low-rank scale adaptation per se.Source: §7 · p. 13

A PROTOCOL DETAILS

• The appendix defines Scale-QLoRA as low-rank per-block scale correction with an on-grid STE and frozen E2M1 codes; QAT-LoRA fake-quantizes full weights, while RTN-merge is the naive re-quantized merge.Four post-hoc quantizers are applied to the same naive adapter without per-quantizer retraining; GPTQ is described as a strong calibrated PTQ baseline.Source: A PROTOCOL DETAILS · p. 15
• All tasks use full training sets, up to three epochs with validation early stopping, and a uniform 5×10−5 learning rate across models, datasets, and methods.Scale ranks are parameter-matched to QLoRA r32; a few retained pilot results use 3k training examples, 500 evaluation examples, and one epoch.Source: A PROTOCOL DETAILS · p. 15
• Main-matrix evaluation uses entire test sets with Wilson 95% confidence intervals around ±0.5–1 pp, while selected Llama and Qwen classification cells use three seeds.Other rows are single-seed; two smaller sub-studies have ±2.6 pp half-widths, and Spider runs differ by 1.84 pp at fixed seed.Source: A PROTOCOL DETAILS · pp. 15–16

B CODE-INVARIANCE: THREAT MODEL AND THE FIVE PAYOFFS IN FULL

• A merged checkpoint can undergo export, conversion/load, and serving transitions that re-derive codes; six NVFP4 outputs from five codebases disagreed across all 224 linears.The six outputs include three harnesses from the authors and three independently shipping tools; same-rule pairs can agree bit-identically on up to 47/224 layers.Source: B CODE-INVARIANCE: THREAT MODEL AND… · p. 16
• On the Banking77 test, fixed per-tensor scalar handling costs −0.81 pp and HuggingFace’s shipped quantize-on-load convention costs −1.04 pp (91.59 vs 92.63).Both are significant under paired McNemar testing (p = 0.002), but these costed divergences are measurements on one task only.Source: B CODE-INVARIANCE: THREAT MODEL AND… · pp. 16–17
• Format conversion re-quantizes every block in both methods’ artifacts; merging a QAST-e4m3 adapter to E8M0 loses 13.0/8.2 pp, so training grid must match deployment.At 120B, applying a deploy rule over a Scale-QLoRA-merged artifact is verified as an identity (max |∆W| = 0).Source: B CODE-INVARIANCE: THREAT MODEL AND… · p. 17
• A QAT-LoRA artifact materialized once with its exact training quantizer and loaded by a code-reading stack serves correctly, with exactly zero differing bytes on the verified path.This safe transition does not extend to format conversion, which re-quantizes both methods’ artifacts.Source: B CODE-INVARIANCE: THREAT MODEL AND… · p. 17
• The sweep partitions rules into amax-derived per-block nearest rounding and rules outside that family, including round-toward-zero, stochastic rounding, and whole-tensor scaling.Rule family alone does not set damage magnitude: scale handling spans observed divergence, and the audit’s costliest divergences are scale conventions.Source: B CODE-INVARIANCE: THREAT MODEL AND… · p. 17

B.1 Payoff 1: the served code plane is invariant under the deploy-rule family

• Across five rules, QAT-LoRA’s best-to-worst spread is 91.6–97.6 pp on Llama-8B and 93.2–97.2 pp on Qwen-30B, while Scale-QLoRA stays at 0.0 pp.At 120B, one RTN-vs-RTZ test verified functional identity for Scale-QLoRA. The NVFP4 sweep lacks a five-rule positive control for scale-merged artifacts, and extreme mismatches are sensitivity bounds, not deployment frequencies.Source: §B.1 · pp. 17–19

B.2 Payoff 2: multi-adapter serving on a shared code plane

• At M=4, Llama-8B scale-space stores 4,992 MB versus 14,976 MB for four QAT-LoRA checkpoints; its in-place scale-byte swap takes 0.059 s versus 7.45 s for re-quantization.Task code bytes are bitwise identical across tasks. The timed weight-space swap re-quantizes; no latency advantage is claimed over an all-resident pointer swap.Source: §B.2 · pp. 19–20

B.3 Payoff 3: invertible merges and rollback

• Subtracting the on-grid scale delta restores the Llama-8B NVFP4 base byte-for-byte, while QAT-LoRA re-derives 7.4% of packed code bytes and cannot recover the base from the merged artifact.The rollback requires the on-grid delta applied at merge, not the raw training delta.Source: §B.3 · p. 20

B.4 Payoff 4: storage deduplication and patch bandwidth

• For four Llama variants, zstd patching is 158 MB versus 3335 MB (21.1×), and 64 KiB chunk deduplication is 416 MB versus 3740 MB (9.0×).QAT-LoRA variants re-derive code planes, so their differences extend across code bytes rather than remaining in the scale plane.Source: §B.4 · p. 20

B.5 Payoff 5: verifiability and provenance

• With codes frozen, any code-byte difference from the base flags tampering or pipeline corruption; the audit is a byte comparison requiring no model execution.QAT-LoRA has no such invariant because its merge changes code bytes by construction.Source: §B.5 · pp. 20–21

B.6 Scope and confounds of the sweep

• Sweep scope varies: Llama uses real vLLM, Qwen uses a single-backend torch harness, and 120B has only one RTN/RTZ pair on CLINC150.The whole-tensor rule is confounded because it changes granularity and collapses even the un-adapted base; the Spider sweep is incomplete.Source: §B.6 · p. 21

C WHY THE NAIVE MERGE RETURNS THE BASE: THE THREE TESTS IN FULL

• On Llama-8B NVFP4, the unadapted block is the global reconstruction optimum on 99.4% of measured blocks; the amax/6 rule returns the block exactly to base on 99.3%.Comparing reconstructed weights matters: stored scales alone would make the premise appear to fail on 83.6% of blocks.Source: C WHY THE NAIVE MERGE RETURNS THE… · p. 22
• On E8M0, weight-MSE finds the base optimal on every block, while the fixed ceiling rule moves 15.5% of blocks off the base and retains adaptation.The E8M0 measurements match the predicted contrast between reconstruction minimization and fixed rounding.Source: C WHY THE NAIVE MERGE RETURNS THE… · pp. 22–23
• For an on-grid base, SSE-optimal reconstruction lands at or just below the un-adapted base; raw losses (−59.8, −38.8, −36.4 pp) mostly track each task’s headroom.Source: C WHY THE NAIVE MERGE RETURNS THE… · p. 24
• On DeepSeek-V4-Flash Spider, RTN retains 101.9% of the gain, while weight-MSE lands within the base interval and retains −6.4%.The result replicates on a second native-FP4 checkpoint and was evaluated against a pre-registered falsifier.Source: C WHY THE NAIVE MERGE RETURNS THE… · p. 24
• The probe is not a census: its comparison uses 16 NVFP4 linears against one E8M0 layer, and it does not explain the NVFP4 accuracy gap.Case 2 covers the scale-and-code reconstruction search over the scale range examined, not every conceivable scale.Source: C WHY THE NAIVE MERGE RETURNS THE… · pp. 25–26

D MATCHED PARAMETERS, TRAINING COST, AND DEPLOYMENT OPTIONS

• On native E8M0 MoEs, Scale-QLoRA merges bit-exactly and QAT-LoRA is accuracy-lossless, but single-seed cells support no accuracy ordering.QAT-LoRA re-derives the code plane; Scale-QLoRA copies codes verbatim.Source: D MATCHED PARAMETERS, TRAINING COST,… · p. 26
• On Llama-8B Banking77, QAST ships +38.7 pp over merged naive QLoRA, while scale adaptation without QAST loses 27.5 pp at merge.The comparison isolates QAST under the full-data protocol.Source: D MATCHED PARAMETERS, TRAINING COST,… · p. 26

D.1 Training cost per step (3.9× dense, 2.2× MoE)

• Memory is model-dependent: Scale-QLoRA uses 49.3 vs 57.7 GB on dense, but 73.6 vs 66.4 GB on MoE.The per-step benchmark does not establish per-run cost because methods may take different numbers of steps to reach their accuracies.Source: §D.1 · pp. 26–27

D.2 Deployment options: zero overhead among the merge-based options measured

• The comparison does not establish an accuracy advantage: Llama-8B matched-parameter results are treated as ties, and Table 5 compares separate adapter runs.QAT-LoRA's merge-quantizer independence depends on its export convention being honored at each code-touching event.Source: Table 4 · p. 27, §D.2 · p. 28
• Among four measured deployment options, Scale-QLoRA alone combines native-4-bit compactness, zero per-forward overhead, bit-exact merge, and merge-quantizer independence within the target format.On Llama-8B NVFP4, the base and merged checkpoint both use 112.76 GiB VRAM; prefill and throughput are within noise.Source: §D.2 · pp. 27–28

D.1 Training cost per step (3.9× dense, 2.2× MoE)

• Scale-QLoRA trains 3.9× faster than QAT-LoRA on the dense model and 2.2× faster on the MoE in the controlled per-step benchmark.QAST fake-quantizes block scales, whereas QAT-LoRA fake-quantizes the full weight each forward pass.Source: §D.1 · p. 26, Table 6 · p. 28

• Scale-QLoRA is the only measured deployment option that combines native-4-bit compactness, zero per-forward overhead, bit-exact merging, and merge-quantizer independence.The other options give up at least one of these properties; the table’s accuracy column is not a controlled comparison.Source: Table 7 · p. 29
• Across merged states and general-capability benchmarks, neither method shows consistent degradation; the worst delta from base is −2.28 pp.These are single runs, so the result is an absence of consistent degradation, not evidence that small forgetting did not occur.Source: E CAPABILITY RETENTION AND GENERATIVE… · p. 28, Table 8 · p. 29
• On MBPP, Scale-QLoRA is bit-exact at 57.6/57.6, while QAT-LoRA is re-quantization-lossless at 73.6/73.6 on Qwen-30B.The dense-model Scale-QLoRA result is +1.4 pp over the bf16 base of 56.2, within the n=500 CI; Qwen-30B is base-saturated, so its exactness result does not demonstrate preserved learning.Source: E CAPABILITY RETENTION AND GENERATIVE… · p. 29

• On gpt-oss, AWQ is indistinguishable from RTN on AGNews and close on Banking77 and CLINC150.Table 16 reports AGNews at −2.6 vs −2.6, Banking77 at +0.1 vs −0.8, and CLINC150 at −2.8 vs −2.6.Source: F WHY THE ACTIVATION-AWARE QUANTIZER… · p. 29
• A shared E8M0 exponent across 32 input channels means within-block-varying AWQ scales can push the exponent up, costing a mantissa bit for all 32 weights.AWQ’s usual per-tensor or per-group deployment does not have this axis collision.Source: F WHY THE ACTIVATION-AWARE QUANTIZER… · p. 29
• On DeepSeek Banking77, per-expert down scales are exactly 1 on 99.85% of experts, while gate_up changes are small and inconsistent across layers.The α search also often selects 0, and block-constant scales remove the measured penalty on nine real expert matrices; the control’s activation statistics are synthetic.Source: F WHY THE ACTIVATION-AWARE QUANTIZER… · pp. 29–30
• DeepSeek AGNews is the lone large E8M0 AWQ exception: −10.8 pp versus RTN’s −3.6 pp, despite slightly worse reconstruction.The other seven measured cells place AWQ within 1.8 pp of RTN; the AGNews deficit does not reproduce on Banking77 under the same batch size, model, and adapter.Source: F WHY THE ACTIVATION-AWARE QUANTIZER… · p. 30
• AWQ scale selection is largely model-dependent rather than calibration-data-dependent on the measured tasks, with Spider a partial exception.Banking77 and CLINC150 share gate_up statistics to four decimals, while Spider has s ≡1 on only 20.9% of gate_up layers.Source: F WHY THE ACTIVATION-AWARE QUANTIZER… · p. 30

G GEOMETRIC SUBSPACE ANALYSIS: WHY ADAPTING ONLY SCALES SUFFICES

• The analysis establishes separate expressivity and no detected accuracy deficit for scale adaptation; the latter is not evidence of equivalence.The results place Scale-QLoRA between PEQA and QLoRA while distinguishing task-equivalent scale solutions from projections of weight-space updates.Source: G GEOMETRIC SUBSPACE ANALYSIS: WHY… · p. 30
• Per-row rescaling is contained in per-block frozen-code scaling, which is contained in unrestricted weight-space adaptation; Scale-QLoRA low-rank-parameterizes the middle family.The scale families preserve the sign and relative code pattern within blocks; PEQA is recovered at granularity G=1.Source: G GEOMETRIC SUBSPACE ANALYSIS: WHY… · p. 30
• On Spider, a fitted continuous scale correction reaches 69.73%, within QAT-LoRA’s 95% interval, but native E4M3 rounding erases it to 61.90%.The fitted relative scale deltas average 0.0037 versus a 0.031 E4M3 relative half-step; a separate learning-rate sweep reverses the method ordering, with 1.84 pp run-to-run spread.Source: G GEOMETRIC SUBSPACE ANALYSIS: WHY… · p. 31
• Lean-4 cross-entropy detects no fitting-capacity deficit for Scale-QLoRA against merge-aware QAT-LoRA, but the comparison is a one-run null, not an effect-size claim.The protocol is lighter than the main matrix; 73% of records are truncated at 4096 tokens, and pass@1 is approximately zero for every configuration in a 50-theorem spot check.Source: G GEOMETRIC SUBSPACE ANALYSIS: WHY… · pp. 31–32, Table 9 · p. 31
• On the tested classification cells, the task needs one or two directions per layer out of 32 trained, and the adapter concentrates 79–86% of spectral energy in one direction.Banking77 completion-masked cross-entropy recovery rises from 0.983 at rank 1 to 1.000 at rank 32.Source: §G.1 · p. 32

G.2 The PEQA↔QLoRA bridge: separate expressivity, not a projection

• Projecting freely trained weight updates onto scale subspaces recovers only a minority of task gain, yet independently trained Scale-QLoRA closes the QLoRA–PEQA gap within the pilot protocol’s noise floor.Full-precision controls on two intent-classification tasks likewise detect no deficit for the per-block multiplicative family, but are single-seed and use an imported noise floor.Source: §G.2 · pp. 32–33, Table 10 · p. 33
• Frozen-code families cannot change a block’s sign or relative pattern, nor move zero-coded weights; 7.1% of NVFP4 weights carry code 0.Raw granularity can briefly help but over-perturbs competent bases; low-rank regularization is needed to manage the extra degrees of freedom.Source: §G.2 · pp. 33–34
• Low-rank scale adaptation closes the measured QLoRA–PEQA gap without establishing an accuracy win, and costs about 1.8× fewer parameters per rank than QLoRA.On CLINC150, Scale-QLoRA rises from 86.8 at r4 to 95.5 at r64; frontier accuracy differences are within noise, making the parameter-cost comparison—not accuracy dominance—the supported result.Source: §G.2 · p. 34

G.3 Capacity, rank curves, and ablations

• At matched parameters, the two lossless methods tie on deployable accuracy; constrained capacity yields a tighter generalization gap when training is stable.The CLINC150 pilot suggests QLoRA's validation-to-test gap is about 1.6× Scale-QLoRA's, but the authors limit the generalization claim to stable training.Source: §G.3 · p. 34
• Rank gains are base-dependent: Scale-QLoRA rises from 81 at r4 to 92.6 at r64 on Banking77, but stops climbing on Spider.Figure 8 reports lossless merging for all three methods on every plotted cell.Source: §G.3 · p. 34, Figure 8 · p. 36
• QAST-on merges with exactly 0.0 pp loss on its target grid, whereas an E4M3-trained adapter merged to E8M0 loses 13.0 pp.The coarser E8M0 grid also lowers unmerged Banking77 accuracy to 86.4 versus 92.8 on E4M3.Source: §G.3 · p. 34, Table 13 · p. 35
• For Llama-8B NVFP4 under the pilot bridge protocol, low-rank Scale-QLoRA closes at least 100% of any real accuracy gap between best QLoRA and PEQA; PEQA-block is no better except Spider (+0.5, within noise).Source: Table 12 · p. 35
• The E4M3 ceiling clamp prevents Qwen-30B's first non-finite event at step 18 and yields merged accuracy 62.6 with zero loss; validation early stopping is the robust tested LR strategy.No fixed learning rate wins both tested regimes.Source: §G.3 · p. 35, Table 14 · p. 35
• At n=500, quantized Scale-QLoRA is indistinguishable from the bf16 LoRA ceiling on Banking77, AGNews, and CLINC150, and trails by up to ∼4 pp on Spider.The CLINC150 and Spider ranges reflect hyperparameter sensitivity, not confidence intervals.Source: Table 15 · p. 37, §G.3 · p. 37

H EXTENDED RELATED WORK

• PEQA fine-tunes frozen-code per-output-channel scales and is a strong floor; LoRDS instead uses continuous low-rank scales and custom kernels, not a standard-format checkpoint.The cited PEQA work is Kim et al. (2023); LoRDS is Tang et al. (2026).Source: H EXTENDED RELATED WORK · p. 37, Merge-aware weight-space… · p. 38

Merge-aware weight-space quantization-aware training (QAT): accuracy-lossless but quantizer-dependent

• QAT-LoRA is accuracy-lossless when merge reproduces its training quantization rule, but it re-derives every E2M1 code rather than preserving the shared code plane.A tool using a different rule can make the QAT-LoRA merge lossy; the paper's comparison to LoTA-QAF remains analytical because it was not tested directly.Source: Merge-aware weight-space… · p. 38

I EXTENDED SCOPE AND DISCUSSION

• The apparent accuracy ordering between Scale-QLoRA and QAT-LoRA is unsupported or fails to survive repetition, so the paper asserts no direction.The contribution is code-invariance and its measured payoffs, not an accuracy ranking.Source: I EXTENDED SCOPE AND DISCUSSION · p. 39
• The guarantee is bit-exact only within a fixed native format, scale grid, block layout, and code plane; cross-grid merging and format conversion fall outside it.A QAST-E4M3 adapter merged to E8M0 loses 13.0/8.2 pp on Banking77/AGNews.Source: I EXTENDED SCOPE AND DISCUSSION · p. 39
• The cross-tool divergence census covers one checkpoint and grid, and the about-a-point divergence measurements cover one task, so neither establishes rates across models, tasks, or formats.Only three of six enumerated output pairs compare tools neither of which is the authors' own harness.Source: I EXTENDED SCOPE AND DISCUSSION · p. 39
• The deploy sweep is a property contrast, not a lifecycle frequency estimate: extreme mismatched-rule results use constructed conditions, and only round-to-nearest is substantiated as shipping behaviour.The scope also includes a lighter pilot protocol for bridge sub-studies and an inconclusive iterated-merge probe.Source: I EXTENDED SCOPE AND DISCUSSION · p. 39
• Measured baselines are the authors' own instantiations, so comparisons with PEQA, QA-LoRA, L4Q, LoTA-QAF, and LoRA-Inlaid are analytical.The sparse-MoE GPTQ caveat concerns guarantees under rank-deficient Hessians, although the paper reports no visible degradation in its matrix.Source: I EXTENDED SCOPE AND DISCUSSION · pp. 39–40
• Generative evidence is effectively one adequately powered task: Spider on DeepSeek; MBPP's 1.4 pp headroom is below its ±4.3 pp Wilson half-width.The authors identify a larger test set and enough training data to open headroom as needed for stronger generative evidence.Source: I EXTENDED SCOPE AND DISCUSSION · p. 40

J BASELINES, AND THREE FURTHER LIMITS OF THE EVIDENCE

• Every measured baseline is an author implementation; comparisons with PEQA, QA-LoRA, L4Q, LoTA-QAF, and LoRA-Inlaid are analytical rather than direct runs of the original methods.A faithful LoTA-QAF comparison requires its own trainer and is left for future work.Source: J BASELINES, AND THREE FURTHER LIMITS… · p. 40
• The divergence census covers one checkpoint, model family, grid, and merged artifact, with only three of six output pairs being tool-against-tool comparisons between independent tools.The about-a-point divergences were measured on one task, and the bridge studies use a lighter pilot protocol.Source: J BASELINES, AND THREE FURTHER LIMITS… · p. 41

K CALIBRATED PTQ ON A SPARSE MOE IS HESSIAN-STARVED

• On DeepSeek-V4 Banking77, 33.7% of gate_up Hessians are rank-deficient and 40 of 11008 experts receive no calibration tokens.The inverse for rank-deficient experts is carried by 1% damping rather than data; the authors attribute this to sparse-MoE calibration under a token budget, not their implementation.Source: K CALIBRATED PTQ ON A SPARSE MOE IS… · p. 41
• A bridge cell has 31.2 pp of fine-tuning headroom and no accuracy ceiling; bridge cells are worst-conditioned because they calibrate from fewer tokens per expert than in-EP cells.The caveat is a limit on calibrated PTQ guarantees for sparse MoE, not doubt about these numbers.Source: K CALIBRATED PTQ ON A SPARSE MOE IS… · p. 41
• GPTQ partially rescues the naive merge on both grids: Llama Banking77 improves from −37.4 to −2.5 pp, and Qwen Banking77 from −23.3 to −0.9 pp.On gpt-oss-120B AGNews, GPTQ's −0.7 pp loss has a 95% CI containing zero.Source: K CALIBRATED PTQ ON A SPARSE MOE IS… · p. 41
• On MXFP4/E8M0, weight-MSE loses −1.2 to −59.8 pp while RTN stays at −3.6 to +0.6 pp; Scale-QLoRA is exactly 0.0 pp on every cell.AWQ tracks RTN on seven of eight cells, with DeepSeek AGNews at −10.8 pp.Source: Table 16 · p. 42
• Across the two grids, reconstruction quality tracks naive-merge collapse directionally, not as a predictor of its size; the weight-space MSE search can fail where RTN is mild.The cited table contrasts RTN, MSE, AWQ, and GPTQ on the same naive adapter without per-quantizer retraining.Source: Table 16 · p. 42