PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantization Error Is Spectrally Flat: A Single Random Probe Is a Calibrated, Data-Free Sensitivity Estimator, with Application to Budget-Targeted Mixed-Precision Quantization

I Kennedy, T Kennedy

arXiv:2609.33923v2cs.CLcs.AIcs.LG

TL;DR

Post-training quantization methods often require representative calibration data, while data-free methods may not target a specified memory budget. RAM uses propagated Gaussian probes and knapsack allocation to choose mixed precision without calibration data. Across seven architectures, it improves median perplexity over size-comparable uniform 4-bit builds on the tested MoE models.

  • Problem

    Calibration-based quantization requires representative data that may be unavailable, mismatched to deployment, or costly to obtain, while threshold-based data-free allocation does not target a chosen memory budget.

    Source: §1 · p. 2

  • Method

    RAM uses propagated Gaussian probes to score tensor quantization at candidate bit-widths, then allocates them under a byte budget with a knapsack solver and guardrails, without calibration data.

    Source: §1 · p. 3, §7 · p. 20

  • Results

    Across seven architectures, RAM builds beat size-comparable uniform 4-bit builds by 3.5 to 13.6% in median perplexity on the tested MoE models.

    Source: §7 · p. 20

  • Takeaways & Limitations

    The probes rank sensitivity by tensor rather than tensor class, and blanket attention-protection and MLP-deprotection rules lose to per-tensor allocation.

    Source: §7 · p. 20

    On Qwen3-8B, RAM trails uniform 4-bit at a tight 4 GB target; calibration data captures activation-dependent sensitivity that random probes do not.

    Source: §6 · p. 19

Abstract

from arXiv · show

A single random Gaussian probe gives an unbiased estimate of the squared Frobenius norm of a layer's quantization error. The estimator is well-behaved because round-to-nearest error is spectrally flat. Across 1,683 tensors from a 35B MoE and a 9B dense model, effective dimensionality is 0.93 to 0.96 times the i.i.d. noise value of the same shape, and on the MoE the median is unchanged from 2-bit to 8-bit. The probe coefficient of variation is predictable from tensor shape. One probe measures per-tensor sensitivity to within 4 to 7%; twenty probes reach 1.3 to 1.4%.RAM applies the propagated form of this estimator to budget-targeted mixed-precision quantization with no calibration data. Gaussian probes carrying the network's own input statistics score every tensor at six bit-widths. A knapsack solver allocates bits under an exact byte budget, with guardrails against catastrophic 2-bit assignments. One probe pass serves any budget. Isolated and propagated scores rank tensors independently on Qwen3.5-35B-A3B (Spearman -0.01), yet the propagated probe rank-correlates 0.81 to 0.83 with the GPTQ layer objective from real activations, while the isolated estimator is uncorrelated with it. That objective is the wrong allocation target: at matched bytes on Qwen3.8-27B, a block-output probe beats a vendor IQ3_M mix and an oracle that allocates from the real-activation objective. On Qwen3-8B the propagated probe ties HAWQ-V2 at matched bytes. Across seven architectures from 8B to 122B, with probe timing up to a 400B model in nine minutes on one workstation, RAM reaches 3.5 to 13.6% lower median WikiText-2 perplexity than size-comparable uniform 4-bit builds on the tested MoE models. (Black Sheep Ai baa.ai)

1 Introduction

RAM addresses the calibration-data and fixed-size limits of existing quantization approaches with random functional probes and budget-targeted mixed precision. Its propagated signal tracks a real-activation layer objective, while matched-budget comparisons show that objective is not always the best allocation target.

1 Introduction

  • Existing calibration-based methods require representative data, while data-free threshold approaches produce one output size rather than targeting a deployment memory budget.Calibration data can be unavailable or mismatched to deployment, and calibration can require hours on high-end hardware for a 400B MoE.Source: §1 · p. 2
  • A single probe estimates squared quantization-error Frobenius norm, while measured spectral flatness supports 4–7% per-tensor sensitivity measurements.Twenty probes reach 1.3–1.4% measurement error.Source: §1 · p. 2
  • RAM scores tensors at six bit-widths with propagated Gaussian probes, then uses a budgeted knapsack allocation with guardrails against catastrophic 2-bit assignments.One probe pass serves any byte budget, and the analysis requires no calibration data.Source: §1 · p. 3
  • On Qwen3.8-27B, a block-output probe beats allocation from the real-activation objective at matched bytes.The propagated probe rank-correlates 0.83 with the GPTQ layer objective on this model when the adaptation rule is switched off, but objective-based allocation still loses.Source: §1 · p. 3
  • Propagated and isolated signals rank tensors independently on Qwen3.5-35B-A3B, but the propagated probe correlates with the GPTQ layer objective under real activations.The propagated correlation is 0.83 on Qwen3-8B, 0.81 on Qwen3.5-9B, and 0.83 on Qwen3.8-27B; the isolated estimator is uncorrelated with that objective.Source: §1 · p. 3
  • Across seven architectures from 8B to 122B, the evaluation covers diverse dense, MoE, FP8-native, and MLA designs, with probe timing measured up to 400B parameters.Quality results span models from 8B to 122B parameters.Source: §1 · p. 3
  • Per-tensor sensitivity ranks attention output/value projections and MoE routers above routed experts, outperforming a blanket rule that protects attention and shared experts.The blanket-protection allocation is both larger and worse than the probe allocation.Source: §1 · p. 3
  • On MLA architectures, compressed key-value projections create sensitivity blind spots for functional probes, a limitation the paper addresses.The paper identifies and resolves this coverage limitation.Source: §1 · p. 3

2 Related Work

Prior quantization work spans calibration-based compression, data-free weight-statistic methods, and sensitivity-based allocation. RAM differs by deriving output-space sensitivity from random inputs and reporting a measured calibration guarantee.

2 Related Work

  • Calibration-based methods use Hessian information, activation-aware scaling, non-uniform quantization, outlier handling, rotations, or codebooks, and require representative calibration data.The cited examples include GPTQ, AWQ, SqueezeLLM, SpQR, QuIP, and AQLM.Source: §2 · p. 3
  • Data-free methods include uniform quantization and weight-statistic sensitivity scores, while several LLM allocators rank layers, experts, or spectral energy.RAM instead scores each tensor by its output-space effect under random inputs and reports a measured calibration guarantee.Source: §2 · p. 3
  • HAWQ variants use calibration-data Hessian-trace sensitivity, while other mixed-precision methods use calibration-based objectives or model-level allocation.GAMMA and MixQuant share RAM’s goal of one analysis for multiple budgets, but use different objectives.Source: §2 · pp. 3–4
  • RAM measures unscaled round-to-nearest error as spectrally flat, unlike LQER’s activation-scaled error shaped for low-rank correction.The paper links this flatness to the accuracy of random probes.Source: §2 · p. 4
  • The Gaussian-probe guarantee is an application of Hutchinson’s trace estimator to the quantization residual, while classical Fisher and Hessian importance measures need data or activations.The paper also relates probe-based rankings to random-projection norm preservation.Source: §2 · p. 4

3 Method

RAM estimates quantization sensitivity using Gaussian probes, first in isolation and then with network-produced inputs and downstream effects. It applies guardrails and a multiple-choice knapsack solver to meet byte budgets, with a specialized probing path for MLA blocks.

3 Method

  • RAM scores candidate quantizations, applies divergence guardrails, and allocates bits using only pretrained weights, without calibration data, gradients, or text forward passes.The method distinguishes an isolated estimator from the propagated version used by the allocator.Source: §3 · p. 4

3.1 Functional Probes on an Isolated Tensor

  • A Gaussian probe’s squared output perturbation estimates quantization-error Frobenius norm; the normalized ratio is consistent but not unbiased.Proposition 3 generalizes the probe to non-identity input covariance and downstream linear maps, so sensitivity can reflect network-produced inputs.Source: §3.1 · pp. 4–5

3.2 Spectral Flatness of Quantization Error

  • Measured quantization error has median effective dimensionality 0.93× the i.i.d.-noise expectation on the 35B MoE and 0.96× on the 9B dense model; 35B medians are 386.1, 386.0, 386.1, and 386.2 at 2, 3, 4, and 8 bits.The paper describes a single random probe as a 4 to 7% measurement.Source: §3.2 · p. 6

3.3 Propagated Probes: The Allocation Signal

  • Propagated probes compare quantized and reference layer outputs using cosine distance, measuring perturbation under network-produced inputs and after nonlinearities and residual paths.An adaptation monitor switches remaining layers to independent probes after sustained large output changes; this triggered on some tested models.Source: §3.3 · p. 6

3.4 Divergence Guardrails

  • Dense models can accumulate small per-tensor divergences into large combined errors, so RAM filters risky 2-bit choices and constrains their allocation.Guardrails include an absolute 2-bit veto, a 30% parameter cap at 2-bit, and a 3.5-bit average floor.Source: §3.4 · pp. 6–7

3.5 Budget-Targeted Allocation via MCKP

  • RAM formulates exact byte-budget allocation as a multiple-choice knapsack problem and greedily upgrades tensors by divergence reduction per added byte.Because scores do not depend on the budget, one probe manifest can be re-solved for multiple target sizes; a share of the budget is reserved for unscored tensors kept at 16-bit.Source: §3.5 · p. 7

3.6 Pipeline Summary

  • The pipeline processes decoder layers sequentially, loading and freeing one layer at a time in lazy mode and writing a per-tensor bit-width manifest for conversion.This permits analysis of models larger than memory.Source: §3.6 · p. 7

3.7 MLA-Aware Probe Coverage

  • For MLA blocks, RAM probes compression weights in hidden space and up-projection weights with latent probes passed through the block’s compression path.On GLM-4.7-Flash, 94 of 701 scored tensors received latent probes; full-dimensional probes remain for weights whose input dimension equals hidden size.Source: §3.7 · pp. 7–8

4 Experiments

The experiments evaluate RAM’s propagated-probe mixed-precision allocation across perplexity and downstream-task outcomes, including how budget, signal, and model affect probe calibration and allocation quality. They also report probe-pass speed relative to rate-distortion analysis and use median per-sequence perplexity as the primary metric.

4.1 Main Results

  • On four MoE models with uniform 4-bit builds within 13% of RAM’s size, RAM lowers median perplexity by 3.5 to 13.6%.The 13.6% gain is on GLM-4.7-Flash; comparisons on Qwen3.5-122B-A10B and Qwen3-8B use larger RAM builds, while the tight-budget Qwen3-30B-A3B allocation remains uniform 4-bit.Source: §4.1 · pp. 8–9, Table 1 · p. 9

4.2 Comparison with a GPTQ-Style Heuristic

  • At 19.5 GB, RAM reaches +1.4% degradation vs BF16, compared with +3.1% for the GPTQ-style heuristic at 20.5 GB.RAM assigns precision per tensor, whereas the heuristic raises all attention and shared-expert tensors to 16-bit and leaves routed experts at 4-bit.Source: §4.2 · p. 9, Table 2 · p. 9

4.3 Probe Convergence Against Exact Ground Truth; 4.4 Calibration of the Propagated Signal

  • At 4-bit, isolated-estimator relative MAE falls with log-log slopes −0.50 and −0.51 on the 35B and 9B models, and 20 probes reach 1.4% error.The single-probe estimate starts at 6% relative error; with 20 probes, rank agreement is ρ = 0.94, with remaining disagreements among tensors whose true sensitivities are tightly clustered. Propagated estimates are less calibrated: 50 sequences yield median errors of 1.0 to 1.7% on the Qwen3.5 models but 19% on Qwen3-8B, whose probe activations vary more.Source: §4.3 · p. 10, Table 3 · p. 10, §4.4 · p. 11

4.5 The Propagated Probe Estimates the Calibration Objective

  • Across three models, propagated probes rank the real-activation objective at 0.83, 0.81, and 0.83, while the isolated estimator is uncorrelated with it.On Qwen3.8-27B, the block-output cosine-and-CKA allocation reaches 5.991 perplexity versus 6.336 for IQ3_M and 6.087 for the real-objective oracle at matched bytes; the oracle loses to cosine-and-CKA on 35 of 40 chunks. The adaptation rule reduces the 27B objective correlation to 0.27, versus 0.83 when disabled.Source: §4.5 · pp. 11–13, Table 5 · p. 12, Table 6 · p. 13

4.6 Dense Model at a Tight Budget: Guardrails and a Negative Result

  • At Qwen3-8B’s 6 GB target, the six-width allocation matches BF16 within 0.2%, while both 4 GB builds overshoot budget and trail uniform 4-bit g64.The 4 GB results motivate a 4-bit minimum: the six-width solver’s 3-bit MLP tensors cost more perplexity than their byte savings.Source: §4.6 · p. 13, Table 7 · p. 14

4.7 Allocating with the Isolated Estimator

  • At the same 19.50 GB budget on Qwen3.5-35B-A3B, isolated and propagated allocations have median perplexities of 6.544 and 6.550, respectively.Their WikiText-2 results are indistinguishable, while the isolated allocation scores 0.679 versus 0.668 on MMLU; it also assigns 109 of 120 expert stacks to 4-bit rather than 74.Source: §4.7 · p. 14, Table 8 · p. 14

4.8 Matched Comparison with HAWQ-V2

  • At matched bytes on Qwen3-8B, propagated probes and HAWQ-V2 have mean WikiText-2 perplexity of 9.641, while the isolated estimator has median 9.737.The propagated and HAWQ-V2 builds are indistinguishable by paired test (p = 0.96); their MMLU scores are 0.749, 0.749, and 0.750 for propagated, HAWQ-V2, and isolated, respectively.Source: §4.8 · p. 15, Table 9 · p. 15

4.9 MLA Probe Coverage

  • On GLM-4.7-Flash at a 16 GB target, the MLA-aware manifest yields a 16.2 GB build, while both isotropic-probe manifests yield 14.6 GB builds.Part of the size gap is due to build size; the supplied passage does not report the corresponding perplexity comparison.Source: §4.9 · pp. 15–16, Table 10 · p. 16
  • MLA-aware probes yield +2.7% degradation versus BF16, compared with +10.4% for the best isotropic-probe manifest and +18.9% for uniform 4-bit.Latent-space isotropic probes spend energy on directions the compression path never produces, so their scores for the 94 post-bottleneck weights misrepresent inference-time behavior; latent probes restrict measurement to the active subspace.Source: §4.9 · p. 16

4.10 Analysis Time

  • The probe pass is 5 to 50 times faster than rate-distortion analysis that quantizes each tensor at 13 bit/group-size configurations in isolation.The alternative was run for three models; Table 11 reports wall-clock analysis time for each model.Source: §4.10 · p. 16

4.11 Median vs. Mean Perplexity

  • Median per-sequence perplexity is the primary metric because a handful of extreme sequences can dominate corpus-level means.On GLM-4.7-Flash, 5 of 128 BF16 sequences exceed 25,000 perplexity, lifting the mean to 11.52 while the median is 8.470; uniform 4-bit has 6 such outliers, mean 14.75, and median 10.075.Source: §4.11 · p. 16

4.12 Downstream Task Evaluation

  • At 33.6 GB, RAM scores 0.731 on MMLU versus 0.721 for BF16 and 0.720 for uniform 8-bit at 34.3 GB; 22.9 GB is the smallest build to beat uniform 4-bit.The 22.9 GB build scores 0.713 versus 0.704 for uniform 4-bit; below that budget, the 19.5 GB and 21.2 GB builds trail it. Every RAM build in the sweep beats uniform 4-bit g64 on median perplexity.Source: Table 12 · p. 17, §4.12 · p. 17
  • Config G’s blanket 8-bit attention rule and Config H’s 4-bit MLP/expert rule both underperform RAM on perplexity and MMLU.Config G scores 6.598 perplexity and 0.645 MMLU versus RAM’s 6.585 and 0.657; Config H scores 6.622 and 0.589. The results indicate that upgrading probe-identified tensors helps more than uniformly adding precision to attention.Source: §4.12 · p. 17
  • At budgets of a third of BF16 and above, RAM dominates uniform quantization on both metrics; below that, it retains lower perplexity, while uniform 4-bit may suit knowledge-intensive tasks.Tight-budget perplexity-optimal allocations push routed expert stacks to 3-bit, whereas MMLU queries every knowledge domain; from about 23 GB upward, expert stacks stay at 4 to 5 bits.Source: §4.12 · p. 17

5 Analysis

The analysis distinguishes isolated weight-error sensitivity from propagated output-space sensitivity, explaining why propagation can rank tensors differently and why sensitivity varies by tensor rather than class. For MoE models, RAM scores fused expert stacks as implementable allocation units.

5.1 Isolated and Propagated Sensitivities Are Different Quantities

  • The isolated estimator ranks tensors by Frobenius error, while the propagated signal reflects how network-produced inputs and downstream paths alter their output impact.On Qwen3.5-35B-A3B, the isolated and propagated signals select different most-sensitive tensors: routers dominate the isolated ranking, while attention projections dominate the propagated ranking.Source: §5.1 · p. 18

5.2 What the Probe Measures That Weight Statistics Do Not

  • Propagation adds network-specific input statistics and measures perturbations after nonlinearities and residual paths, allowing equal-Frobenius-error tensors to receive different scores.The same mechanism applies inside a block when probes pass through the compression path, as in the MLA fix.Source: §5.2 · p. 18

5.3 Which Tensors Are Sensitive

  • MoE sensitivity varies by individual tensor: routers and attention rank high while routed experts rank low, but MLP projections rank highest among Qwen3-8B’s dense tensors.On Qwen3.5-35B-A3B, 93 of 260 attention tensors receive 16-bit while 22 receive 2-bit, and routed expert stacks span four bit-widths.Source: §5.3 · pp. 18–19

5.4 MoE Expert Handling

  • RAM scores each fused MoE (layer, projection) expert stack as one tensor, with divergence measuring the propagated effect of quantizing all experts in that stack together.On Qwen3.5-35B-A3B, 120 stacks cover 32.2 of the model’s 33.6 billion scored parameters.Source: §5.4 · p. 19

6 Limitations and Discussion

RAM’s results are bounded by the tested models, quantizer, and evaluation tasks. Its propagated signal can be poorly calibrated per sequence on one model, and equal objective cost does not guarantee equal end-to-end quality.

RTN base quantizer

  • The model-level quantizer comparison has one data point: HQQ improves BF16-evaluated Qwen3-8B perplexity by 2.2%, but packed-weight conversion did not preserve that gain.Combining a better per-tensor quantizer with mixed-precision allocation remains open; the authors make no claim that RTN is optimal.Source: §6 · p. 19

Isotropic probes and the propagated signal

  • On one of three models, the propagated signal is not calibrated per sequence; its variance depends on input second moments and the downstream Jacobian.The Gaussian probe distribution is a modeling choice, and the hybrid 27B estimate requires disabling the adaptation rule.Source: §6 · p. 19

Rank swaps and end-to-end quality

  • Equal total divergence can yield different perplexity, and the authors have not measured perplexity spread across probe seeds for the reported builds.The allocation test covers one model and one quantizer family; MoE expert-stack inputs were not measured.Source: §6 · pp. 19–20

Dense model gains

  • RAM’s dense-model evidence is limited to Qwen3-8B, where it matches BF16 at 48% of its size but loses to uniform 4-bit at a tight budget.The authors attribute the allocator’s room to work partly to MoE sensitivity heterogeneity; dense models have more uniform sensitivity and a smaller vocabulary-relative budget for unquantized embeddings.Source: §6 · p. 20

Downstream evaluation

  • WikiText-2 perplexity and MMLU were evaluated on one model family, so they do not establish behavior on procedural or agentic tasks.The authors call for broader multi-task and behavioral evaluation.Source: §6 · p. 20

7 Conclusion

RAM uses Gaussian probes to estimate quantization sensitivity and allocate mixed precision without calibration data. Across seven architectures, its MoE builds improve on size-comparable uniform 4-bit builds, while sensitivity patterns and median perplexity also yield broader observations.

Conclusion

  • Gaussian probes estimate squared Frobenius quantization error, while measured spectral flatness makes single-probe sensitivity accuracy predictable from tensor shape.Propagated probes apply the estimator to an input- and downstream-weighted error operator; a knapsack solver converts one probe pass into an allocation for any byte budget.Source: §7 · p. 20
  • 3.5 to 13.6% lower median perplexity than size-comparable uniform 4-bit builds on tested MoE models across seven architectures.RAM also beats a GPTQ-style heuristic at a smaller size and reaches BF16-level MMLU at half the BF16 size.Source: §7 · p. 20
  • Probe rankings show tensor-specific sensitivity, and median per-sequence perplexity is safer than the mean because a few outlier sequences can dominate it.The outliers can be near-random even for the BF16 model.Source: §7 · p. 20

A Proof Details and Variance Bounds

The appendix derives variance and concentration bounds for Gaussian-probe estimates, with effective dimensionality governing how tightly a probe approximates the error norm. The predicted coefficient of variation agrees with observed values, and typical tensors are estimated within about 10% by one probe with high probability.

A.1 Variance of the Probe Estimator

  • The coefficient of variation falls as 1/√p when using p probes, and the section derives a variance bound for Gaussian probes using the fourth-moment identity.The bound is stated for x ∼ N(0, I_n) and A = ∆⊤∆.Source: §A.1 · p. 23
  • On Qwen3.5-35B-A3B, the predicted CV is about 7.2% for one probe and 1.6% for 20; on Qwen3.5-9B, it is 3.7% and 0.8%, respectively.The predictions match empirical CVs with median ratios of 0.999 and 1.000.Source: §A.1 · p. 23

A.2 Concentration and the Role of Spectral Flatness

  • Quadratic-form concentration bounds control deviations of the probe estimate in terms of the spectrum of A = ∆⊤∆, with effective dimensionality acting as the sample size.An ambient-dimension Johnson–Lindenstrauss-style bound would be valid only for exactly isotropic error.Source: §A.2 · p. 23
  • For typical effective dimensionality of about 400 to 1,500, one probe gives a (1 ± 0.1)-approximation with high probability, consistent with observed 4 to 7% per-probe CVs.The concentration bound depends on the error spectrum through effective dimensionality, not merely the ambient dimension.Source: §A.2 · p. 23

A.3 Effective Dimensionality of Quantization Error: Measured

Direct measurements refute an earlier conjecture that quantization error has low effective dimensionality: it is near the i.i.d.-noise ceiling for most tensors. A small set of structured tensors falls well below that ceiling and can be identified from probe-time sample variance.

A.3 Effective Dimensionality of Quantization Error: Measured

  • The measurements refute an earlier conjecture of deff ≪ min(m, n) and the estimate deff ≈30 to 50 inferred from an apparent faster-than-√p convergence slope.The section attributes that slope artifact to an explanation in §4.3.Source: §A.3 · p. 23
  • Group-wise round-to-nearest error is close to white noise: effective dimensionality reaches 0.93 of the i.i.d.-noise ceiling for MoE expert stacks and 0.96 for dense tensors.Outliers and inter-group correlation reduce the noise ceiling by only a few percent overall.Source: §A.3 · pp. 23–24
  • A small tail of structured tensors, including 32-row linear-attention projections and one vision-tower output projection, falls well below the i.i.d.-noise ceiling.These are the tensors for which more probes are needed; multi-probe sample variance estimates each tensor's CV at no additional cost and identifies them during probing.Source: §A.3 · p. 24

B Alternative Per-Tensor Quantizers

The tested alternatives show that better rounding can reduce reconstruction error, but some methods require calibration data or runtime support, and gains may not survive format conversion. In these experiments, bit allocation mattered more than per-tensor rounding, though a better rounder could add a further percent or two at model level.

B Alternative Per-Tensor Quantizers

  • HQQ lowered per-tensor NRMSE by 2.96% on average across 30 tensors; on Qwen3-8B, BF16 evaluation reached 9.768 median perplexity versus 9.987 for RTN, but runtime re-rounding yielded 10.338.Integrating HQQ into the pipeline requires a format-aware conversion that was not built.Source: B Alternative Per-Tensor Quantizers · p. 24
  • A random orthogonal rotation lowered per-tensor NRMSE by 7.92% on average across all 30 tensors, but model-level evaluation was omitted because deployment required unsupported custom dequantization kernels.The required kernel computes y = Wq(H⊤x).Source: B Alternative Per-Tensor Quantizers · p. 24
  • Data-free OBQ worsened NRMSE by 0.50% and improved 0 of 18 tensors, consistent with its weight-only Hessian being a poor proxy for the activation Hessian.The passage says OBQ needs calibration data.Source: B Alternative Per-Tensor Quantizers · p. 24
  • Data-free AdaRound changed NRMSE by exactly 0.000%, because weight-only MSE selects round-to-nearest, which RTN already computes.Source: B Alternative Per-Tensor Quantizers · p. 24
  • Where bits are allocated mattered more than how each tensor was rounded, though HQQ could add a further percent or two at model level.The paper reports this as an experimental comparison.Source: B Alternative Per-Tensor Quantizers · p. 24

C Allocation by Tensor Class

Table 14 summarizes RAM build manifests by tensor class for three models and their target sizes. The listed classes group attention projections, routed expert stacks, and other qualifying tensors.

C Allocation by Tensor Class

  • Table 14 summarizes allocation by tensor class for Qwen3.5-35B-A3B at a 21 GB target, GLM-4.7-Flash at 16 GB, and Qwen3-8B at 6 GB.The GLM-4.7-Flash build is MLA-aware.Source: C Allocation by Tensor Class · p. 24
  • “Attention” includes attention and linear-attention projections, while “Routed experts” denotes fused per-layer expert stacks.Source: C Allocation by Tensor Class · p. 24
  • “Other” covers small convolution and gating tensors above 1,024 elements; embeddings, language-model heads, and norms are not listed.Source: C Allocation by Tensor Class · p. 24

D Evaluation Protocol

The evaluation reports mean and median perplexity on WikiText-2 under a fixed sequence protocol, with two community builds using a different sequence count. The section also specifies hardware, quantization settings, and table conventions.

D Evaluation Protocol

  • WikiText-2 test perplexity uses 128 sequences of 2,048 tokens, seed 42, and no sliding window, except two community uniform builds in Table 1 evaluated on 256 sequences.Each sequence is evaluated independently.Source: D Evaluation Protocol · p. 24
  • Median perplexity is the 50th percentile of per-sequence perplexities, while mean perplexity is the exponential of average cross-entropy across all tokens.Per-sequence perplexity is the exponential of mean cross-entropy loss.Source: D Evaluation Protocol · p. 24
  • Table 15 reports 4-bit effective dimensionality against the i.i.d.-noise prediction, predicted and empirical probe CVs, and the median ratio of probe mean to exact error norm.Table 14 instead summarizes allocation manifests by tensor class; embeddings, language-model heads, and norms are omitted.Source: Table 15 · p. 25, Table 14 · p. 25
  • Experiments ran on one Apple Mac Studio with an M2 Ultra, 192 GB unified memory, a 24-core CPU, and a 76-core GPU; no distributed setup was required.Source: D Evaluation Protocol · p. 25
  • Quantization uses group-wise round-to-nearest with affine scaling, usually group size 64, packed integer storage, and a 16-bit scale and bias per group.Source: D Evaluation Protocol · p. 25

E Direct Measurement of Quantization-Error Spectra

Direct measurements find that round-to-nearest quantization error is close to the effective dimensionality predicted for i.i.d. noise, across tensor types and bit-widths. The section also provides measurement and reproduction details for the reported analyses.

E Direct Measurement of Quantization-Error Spectra

  • For weight tensors with both dimensions at least 32, excluding embeddings and the LM head, the protocol computes float32 RTN error at 2, 3, 4, and 8 bits with group size 64.Source: E Direct Measurement of… · p. 25
  • For Qwen3.5-35B-A3B at 4-bit, fused MoE expert stacks had effective dimensionality 2,031.7 versus a noise prediction of 2,032.1.Expert down-projections measured 511.5 versus 511.5.Source: E Direct Measurement of… · p. 25
  • The largest deviations were small or structurally atypical tensors: 32-row linear-attention a/b projections had effective dimensionality about 25 versus 31.5, and one vision-tower MLP output projection had 83 versus 909.The passage says multi-probe sample variance can detect these cases.Source: E Direct Measurement of… · p. 25
  • Median effective dimensionality stayed nearly unchanged from 2 to 8 bits, while MAE convergence slopes ranged from −0.49 to −0.51 at every bit-width on both models.For the 35B model, medians were 386.1/386.0/386.1/386.2 at 2/3/4/8 bits; for the 9B model, they were 1,469/1,424/1,476/1,457.Source: E Direct Measurement of… · pp. 25–26

F Reproduction Instructions

  • The reproduction instructions identify scripts for probe analysis, model conversion, and perplexity evaluation, with published pre-quantized models and specified software versions and hardware.The probe-analysis command uses 50 propagated probes, six bit-widths, and group size 64; the documented machine is an M2 Ultra Mac Studio.Source: F Reproduction Instructions · p. 26
Loading 2609.33923v2…