PaperlayerView exact revision on arXiv

Source-linked AI summary

PrismQuant: Optimal Null-Space Rotations for Grouped Quantizers

Yanlong Chen, Yining Chen, Song Zhang, Amirhossein Habibian, Yawei Li

arXiv:2609.32429v1cs.AIcs.LG

TL;DR

Activation anisotropy can let a few channels or low-dimensional directions dominate the quantization range, while magnitude alone does not determine quantization difficulty because alignment with the quantizer matters. PrismQuant aligns leading activation eigendirections with the group-constant subspace represented by asymmetric grouped-quantizer offsets, using a closed-form spectral optimum and compact training-free Householder transforms. Among compared methods, PrismQuant’s strongest configuration reaches the lowest 8.58 WikiText-2 perplexity and the highest eight-task zero-shot mean (61.23%).

  • Problem

    Activation anisotropy can let a few channels or low-dimensional directions dominate the quantization range, while magnitude alone does not determine quantization difficulty because alignment with the quantizer matters.

    Source: §1 · pp. 1–2

  • Method

    PrismQuant aligns leading activation eigendirections with the group-constant subspace represented by asymmetric grouped-quantizer offsets, using a closed-form spectral optimum and compact training-free Householder transforms.

    Source: §1 · p. 2

  • Results

    Among compared methods, PrismQuant’s strongest configuration reaches the lowest 8.58 WikiText-2 perplexity and the highest eight-task zero-shot mean (61.23%).

    Source: §1 · p. 3

  • Takeaways & Limitations

    The quantizer’s existing affine offsets can represent aligned shared levels without widening within-group range, making the format’s geometry a target for rotation design.

    Source: §1 · p. 2

    The range predictor is an approximation, not a bound, and assumes the mixed residual keeps its normalized shape as energy is removed.

    Source: §3.5 · p. 8

Abstract

from arXiv · show

Smaller activation outliers do not necessarily imply better low-bit quantization: their alignment with the quantizer matters. We introduce PrismQuant, a quantizer-aware rotation framework that aligns the leading activation eigenspace with the constant group subspace of asymmetric grouped INT4. The affine offsets represent the energy in this subspace without widening the range within the group. We formulate rotation design as a Ky Fan trace maximization and derive a closed-form solution that is provably optimal for this alignment objective. Compact Householder transformations and their compact-WY representation enable gradient-free construction and efficient application at both foldable and online sites. A predictive range law further connects unaligned activation energy and group size to quantization-relevant variation. Experiments on Llama, Qwen, and Mistral span dense models up to 70B parameters and a 30B mixture-of-experts model. Under W4A4KV4, PrismQuant sets the state of the art on Llama-3.2-3B among the compared methods in both perplexity and accuracy. On Llama-3.1-70B, it attains 3.85 perplexity and 72.46% average zero-shot accuracy, only 0.22 percentage points below full precision. In the deployment study on Llama-3.1-8B, our optimized implementation achieves 1.51x prefill and 1.22x CUDA Graph decode speedups over matched FP16 baselines, with 56.34% lower decode peak memory and only 2.35% additional Graph decode latency over Hadamard. Code is available at https://github.com/ForeverBlue816/PrismQuant.

1 INTRODUCTION

PrismQuant targets the group-constant directions that asymmetric grouped quantizers represent through their affine offsets, aligning dominant activation energy with those directions. Its reported results span local quantization behavior, language-model benchmarks, and group-size trade-offs.

1 INTRODUCTION

  • A group-constant component can be represented by the asymmetric quantizer’s affine offset without increasing within-group range, making quantizer geometry—not magnitude alone—relevant to quantization difficulty.The introduction frames activation anisotropy as a challenge because dominant directions can leave insufficient resolution for the remaining signal.Source: §1 · pp. 1–2
  • The method aligns leading activation eigendirections with the quantizer’s group-constant subspace using a closed-form spectral optimum and compact, training-free Householder transforms.The transforms fold into adjacent weights where possible; non-foldable sites use a compact online representation.Source: §1 · pp. 2–3
  • At a matched activation-bit budget, tested configurations favor finer groups over adding affine directions within larger groups, so alignment capacity and quantization granularity must be considered together.The paper derives a range law separating residual unaligned energy from group-size dependence under a Gaussian residual approximation.Source: §1 · p. 2
  • In end-to-end comparisons, PrismQuant’s strongest configuration achieves the lowest WikiText-2 perplexity (8.58) and highest eight-task zero-shot mean (61.23%) among compared methods.Source: §1 · p. 3

2 RELATED WORK

Prior quantization work reshapes activation distributions or treats selected outlier directions specially, while PrismQuant instead targets directions already represented by grouped asymmetric quantizers. Related work also connects grouping to quantization granularity and metadata.

2 RELATED WORK

  • Channel-wise smoothing and scaling methods rebalance ranges without mixing information across channels, while token-level isolation is described as complementary to PrismQuant.The passage names SmoothQuant, OmniQuant, Outlier Suppression+, AWQ, and PrefixQuant.Source: §2 · pp. 3–4
  • Rotation-based methods including QuaRot, SpinQuant, and DuQuant primarily optimize activation geometry.Other cited methods further optimize scaling, rotation geometry, or activation distributions.Source: §2 · p. 4
  • ResQ (Saxena et al., 2024) preserves high-variance directions at higher precision and OffQ (Wang et al., 2026) targets token outliers, whereas PrismQuant treats the quantizer’s group-constant subspace as the alignment target.The paper presents PrismQuant’s rotation as a closed-form spectral optimum over the full second moment.Source: §2 · p. 4
  • Grouped asymmetric quantizers use scales and zero-points; their group size controls local resolution, metadata cost, and the dimension of the subspace available for alignment.KIVI (Liu et al., 2024) is cited as showing that quantization granularity should reflect the statistical structure of keys and values.Source: §2 · p. 4

3 METHOD

PrismQuant exploits the group-constant directions represented by asymmetric quantizer offsets, aligning leading activation energy with them through an optimal spectral rotation. The method constructs compact transforms, integrates them at Transformer sites, and analyzes how residual energy and group size relate to range and metadata.

3 METHOD

  • PrismQuant targets the quantizer’s group-constant, range-neutral subspace, steering dominant activation energy into levels already represented by the affine offset.The paper then derives spectral alignment, compact Householder transforms, Transformer integration, and the metadata trade-off.Source: §3 · p. 4

3.1 QUANTIZER-INDUCED RANGE-NULL SUBSPACE

  • For each group, the affine offset represents a shared level while the within-group range depends on the remaining variation; the group-constant subspace therefore contributes no within-group range.Its degrees of freedom are represented through offsets already stored by the format, rather than through explicit mean subtraction.Source: §3.1 · p. 5

3.2 OPTIMAL ALIGNMENT

  • The rotation objective maximizes activation energy projected onto selected group-constant targets, and the Ky Fan principle gives an optimum by mapping a leading eigenspace onto those targets.At k = M this minimizes total energy outside the target subspace; for k < M the guarantee applies to the selected k-dimensional target.Source: §3.2 · pp. 5–6

3.4 DEPLOYMENT IN THE TRANSFORMER

  • PrismQuant folds rotations into weights at many sites, while the down-projection input retains an online block Hadamard and rank-k correction; value rotations align the V cache.Residual-stream rotations are folded into weights, whereas a rotation after the element-wise gate cannot be moved through that gate.Source: Figure 3 · p. 6, §3.4 · p. 7

3.3 STRUCTURED CONSTRUCTION

  • A compact-WY Householder construction maps leading eigendirections to coordinate anchors, then uses a permutation, signs, and block Hadamard transform to produce group-constant directions.The rank-k correction costs O(dk) per token, and the complete transform costs O(dk + d log g).Source: §3.3 · pp. 6–7

3.5 RANGE AND METADATA

  • The range predictor relates residual unaligned energy and group size to quantization step, while smaller groups jointly increase resolution and alignment capacity at greater metadata cost.The predictor is an approximation, not a bound, and its shape factor is measured near one at q/k/v inputs and 0.87–0.92 at down-projection inputs.Source: §3.5 · p. 8

4 EXPERIMENTS

The experiments evaluate W4A4KV4 quantization across dense Llama models, Qwen3-30B-A3B-Base, and additional Qwen and Mistral models. They compare PrismQuant with Hadamard and other methods, then examine alignment rank and other design choices.

4 EXPERIMENTS

  • The evaluation uses W4A4KV4 post-training quantization, with asymmetric group-quantized activations, GPTQ INT4 weights, and a KIVI KV cache.Models include Llama-3.2-3B, Llama-3.1-8B and 70B, Qwen3-30B-A3B-Base, Qwen3 dense models, and Mistral-7B-v0.3.Source: §4 · p. 8

4.1 DENSE MODELS

  • 72.46% average accuracy on Llama-3.1-70B leaves PrismQuant 0.22 points below bf16; it also has the lowest perplexity among displayed quantized methods across three Llama scales.The comparison covers WikiText-2 perplexity and eight-task average zero-shot accuracy.Source: §4.1 · p. 8, Table 1 · p. 9

4.2 MIXTURE OF EXPERTS

  • At k = max, PrismQuant recovers 46% and 38% of the two perplexity gaps; at k = 8, it recovers 1.05 of 1.24 accuracy points and beats Hadamard on all eight tasks.The k = 8 result is 0.19 points below bf16; the two PrismQuant rows share expert rotations and differ only in R1 rank.Source: §4.2 · p. 9, Table 2 · p. 9

4.3 ABLATION STUDIES

  • Rank recovery is not monotone in induced perplexity despite exact alignment at each k; further ablations probe metadata budgets, cold-expert calibration, and calibration-token choice.On Qwen3-8B, recovery rises from 59.12% to 99.95%.Source: §4.3 · pp. 9–10, Table 3 · p. 10

5 CONCLUSION

PrismQuant uses a closed-form spectral alignment objective and a rank-k Householder realization, with a range law relating residual unaligned energy and group size. The conclusion identifies deployment overhead and the limits of quantization evaluations as relevant boundaries.

5 CONCLUSION

  • PrismQuant reduces rotation design to a closed-form spectral optimum and a rank-k Householder realization, and the alignment extends to formats scaled by group extrema.The conclusion says the construction is insensitive to residual permutation, calibration-set choice, and eigenspace-solution precision.Source: §5 · p. 10
  • A native INT4 GEMM for group-wise asymmetric activations is needed to turn the transform’s measured overhead into a checkpoint-level system.Source: §5 · p. 10
  • The authors used generative AI for drafting, figure preparation, and language editing, and manually verified drafted text, figures, and citations.They report not using it to generate research ideas, experimental code, or calibration and benchmarking data.Source: AI USE STATEMENT · p. 11
  • The study covers existing models and public benchmarks, and improved quantization accuracy is not evidence of model safety or fairness.The conclusion calls for applicable license compliance and application-specific safety assessments.Source: ETHICS STATEMENT · p. 11
  • The implementation records experimental configurations and includes numerical checks of rotation equivalence and structured-transform correctness.Source: REPRODUCIBILITY STATEMENT · p. 11

A.1 EXPERIMENTAL PROTOCOL

The evaluation applies a shared quantization protocol across dense and routed models, then reports model-family comparisons against matched Hadamard and selected published rows.

A.1 EXPERIMENTAL PROTOCOL

  • W4A4KV4 uses dynamic asymmetric INT4 activations (g = 128), GPTQ asymmetric INT4 weights, and KIVI-quantized keys and values, with the latest 32 tokens kept full precision.The model sweep spans Llama, Qwen3 Base, Qwen3-30B-A3B-Base, and Mistral; weights use per-channel quantization for Llama-3.1-8B and 70B and per-group quantization for the other listed models.Source: §A.1 · p. 15
  • Hadamard and PrismQuant share the pipeline settings except for the rotation, and end-to-end rows use three seeds.Evaluation includes WikiText-2 and C4 perplexity plus zero-shot accuracy; calibration uses 128 WikiText-2 training sequences of 2,048 tokens.Source: §A.1 · p. 15
  • For Qwen3-30B-A3B, each expert uses k = 6 and its own GPTQ Hessian; 1,346 experts with fewer than 512 routed calibration tokens use layer-pooled Hessians, and fused-kernel timing is omitted.The router remains in bf16 and reads unquantized post-norm activations; the expert input is quantized once per token and shared by its eight routed experts.Source: §A.1 · pp. 15–16

A.2 QWEN3 DENSE FAMILY

  • At k = 8, PrismQuant exceeds Hadamard on Qwen3 MMLU by 0.7–2.4 points and GSM8K by 2.6–4.4 points at every size.Qwen3-8B is exceptional: PrismQuant removes 64% of Hadamard’s 29% WikiText-2 degradation (11.39 → 9.73), where most down-projection input lies on a group-constant direction.Source: Table 4 · p. 16, §A.2 · p. 16

A.3 MISTRAL-7B-V0.3

  • 5.62 WikiText-2 perplexity on Mistral-7B-v0.3 is below every displayed entry, while SpinQuant retains the higher reported eight-task mean (68.60%).PrismQuant’s eight-task gain is 0.38 points, recovering 26% of Hadamard’s loss; the accuracy margin is within one standard error.Source: §A.3 · p. 16, Table 5 · p. 17

A.4 ADDITIONAL EXPERIMENTS

The additional experiments test metadata allocation, expert calibration, token selection, and robustness; the algorithm and precision analysis clarify how the rotation is constructed and how its group-constant levels are represented.

A.4 ADDITIONAL EXPERIMENTS

  • At a shared 4.25-bit budget, the default (128, 1) configuration beats (256, 3) on both models, so halving group size is worth more than adding two directions.At 4.13 bits, (256, 1) already matches or beats Hadamard at (64, 1); extra directions help PrismQuant only when energy is aligned into them.Source: §A.4 · p. 17
  • Cold-expert calibration choices differ by at most 0.023 PPL, far less than the 0.26 margin over Hadamard, so alignment on well-populated experts carries the gain.The calibration set contains 2,002 of 6,144 experts with fewer than 2,048 routed tokens; shrinkage is best among the compared schemes.Source: §A.4 · pp. 17–18
  • The all-token estimator has the lowest perplexity on both models at every site, while top-1% token selection falls between it and top-1 selection.Outlier-token selection relies on BOS; removing BOS costs another 0.04–0.08 PPL and reaches or exceeds Hadamard, while the top-1 subspace still recovers about three quarters of the all-token gain.Source: §A.4 · pp. 18–19, Table 7 · p. 18
  • Changing calibration corpus reassigns 99% of residual coordinates and lowers down-projection captured energy from 0.315 to 0.273, yet perplexity barely moves; three eigensolver passes match exact decomposition within seed noise.A single pass captures f = 0.009 and costs 0.059 PPL, while two passes recover f = 0.290.Source: §A.4 · p. 19, Table 8 · p. 19
  • At g = 128, PrismQuant’s margin over Hadamard is unchanged under symmetric per-group INT4, while moving anchors to a nonconstant Walsh column costs 16–25% of the gain under asymmetric quantization.With one offset per token (g = d, k = 1), the offset accounts for 37% of the gain on Qwen and 69% on Llama; the anchor-group range grows only 11–19% when the level enters the range.Source: §A.4 · pp. 19–20, Table 9 · p. 20

A.5 ALGORITHM

  • The algorithm estimates site-specific leading eigendirections, maps them to group anchors with compact-WY Householder reflections, and uses a signed permutation and block Hadamard to complete the rotation.It folds R1 and R2 into weights, registers online R4 as a rank-k correction, then applies GPTQ to weights and attaches activation and KV quantizers.Source: §A.5 · pp. 20–21

B.1 GROUP-CONSTANT GEOMETRY AND METADATA PRECISION

  • In ideal affine quantization, a group-constant level is represented by the offset without increasing range, but fp16 metadata means arbitrary-shift code invariance is not exact.The implementation stores fp16 scale and offset; the paper distinguishes this rounding error from the ideal real-arithmetic range identity.Source: §B.1 · p. 21

I. Spectral alignment: Householder / compact-WY

PrismQuant constructs a rank-controlled orthogonal rotation by aligning leading eigendirections to group anchors, then uses it in a grouped INT4 quantization pipeline. The aligned level is carried by the affine offset, whose finite precision is separately assessed.

I. Spectral alignment: Householder / compact-WY

  • The site-specific rank is capped by the number of group slots, and compact-WY Householder factors map the selected eigendirections to distinct group anchors before block Hadamard mixing.The remaining coordinates are permuted and signed; the correction has rank at most k.Source: §I · p. 22
  • R1 and R2 are folded into surrounding weights, while online R4 is a rank-k correction inside the down-projection input; R3 is the identity because keys are quantized per channel without rotation.The pipeline then quantizes rotated weights with GPTQ and attaches activation and KV quantizers.Source: §I · p. 22
  • Grouped asymmetric INT4 represents a group-constant level through its offset rather than its scale, preserving the ideal range benefit of the alignment.The quantizer uses an fp16 scale and offset.Source: §I · p. 22
  • At fixed codes and scales, fp16 offset rounding has median error 0.001 group steps and changes perplexity by less than 0.001 when re-stored as fp32 or bf16.The offset is not lossless, but the measured rounding error is far below the alternative unaligned range cost.Source: §I · p. 23, Table 11 · p. 23

B.2 PROOF OF OPTIMAL SPECTRAL ALIGNMENT

The proof shows that mapping leading activation eigendirections into the selected group-constant target maximizes captured energy. The construction realizes that alignment with compact Householder factors, while the range predictor adds assumptions beyond the spectral guarantee.

B.2 PROOF OF OPTIMAL SPECTRAL ALIGNMENT

  • The Ky Fan maximum principle bounds captured energy by the sum of the k largest eigenvalues, with equality when the transformed target spans the leading eigenspace.For k below the full target dimension, optimality applies to the selected k-dimensional target; eigenspace estimation and empirical calibration remain distinct from the population optimum.Source: §B.2 · p. 23

B.3 HOUSEHOLDER CONSTRUCTION AND COMPACT REPRESENTATION

  • Sequential Householder reflections preserve previously aligned anchors, and the signed-permutation-plus-Hadamard construction maps selected directions to group-constant vectors.The compact-WY representation applies the correction as Gx = x − W(Y^T x), with at most k columns.Source: §B.3 · p. 24

B.4 CALIBRATION AND NUMERICAL APPROXIMATION

  • The randomized eigensolver uses three passes at wide sites, while calibration moments and rotations are site-specific and fixed before evaluation without gradient-based training.The resulting calibration optimum is not guaranteed to be optimal for unseen activations.Source: §B.4 · p. 24

C.1 RANGE PREDICTION

  • The deterministic residual-energy inequality tightens a range bound but does not guarantee a decrease in every realized group range.Predicting typical range requires an additional residual-distribution model.Source: §C.1 · pp. 25–26

B.5 GAUSSIAN RANGE MODEL AND THE TWO-FACTOR PREDICTOR

  • Under a Gaussian residual model, the group-size factor is an expected range term; the finite-g values are η64 ≈ 4.687467, η128 ≈ 5.189195, and η256 ≈ 5.653727.Comparing group sizes also assumes comparable Gaussian scale parameters.Source: §B.5 · p. 25, §C.1 · p. 26

C.1 RANGE PREDICTION

  • The baseline-relative range predictor assumes comparable residual shapes and scales, effectively taking the PrismQuant and Hadamard crest-factor ratio as one and Hadamard’s captured fraction as zero.Unused slots, correlations, heavy tails, nonuniform scales, and calibration mismatch can change that ratio; spectral optimality does not imply the residual-shape assumption.Source: §C.1 · pp. 26–27

C.2 METADATA ACCOUNTING AND EXTENDED-AFFINE CONTROLS

The section compares activation representation choices under metadata constraints, distinguishing finer grouping from adding explicitly represented directions. Extended-affine controls test how those resources trade off at a shared bit budget.

Two resources, one budget

  • Halving group size doubles both local scales and constant directions, while raising rank leaves the format budget unchanged but adds factor storage and online work.Codes and one FP16 scale and offset per group occupy 4 + 32/g bits per value; extended-affine controls with m −1 added directions use 4+16(m+1)/g bits.Source: §C.2 · p. 27

Metadata precision

  • Rounded FP16 scales and offsets can change integer codes from the ideal grid, although a shared level remains exactly range-neutral.The stated bound assumes finite rounded parameters, positive scale, and otherwise exact arithmetic; additional floating-point errors are outside it.Source: §C.2 · p. 27

Extended-affine controls

  • Extended-affine controls add m −1 FP16 coefficients along fixed, orthonormal, nonconstant Walsh directions, separating represented capacity from scale resolution.The explicitly represented subspace has dimension m; unlike the original group-constant direction, the added directions generally have nonzero range.Source: §C.2 · p. 27

Measured comparisons

  • At 4.25 bits per value, (g, m) = (128, 1) outperforms (256, 3) in activation-only perplexity on both tested Llama checkpoints.PrismQuant at (256, 1) also outperforms Hadamard at (128, 1) while using fewer logical activation bits.Source: §C.2 · pp. 27–28

Range diagnostics

  • Figure 4's pooled range observations include activation, V-cache, and extended-affine controls, while its held-out Qwen3 expert panel shows wider scatter below 2,048 calibration tokens.The panel's holdout is for the pooled regression, not the experts' own rotation calibration.Source: §C.2 · p. 28, Figure 4 · p. 28

C.3 EMPIRICAL DIAGNOSTICS OF THE RANGE LAW

The range-law diagnostics test the pooled trend, group-size factor, and per-site predictions, including their limits. The deployment sections then select a low-rank operating point and describe compact online execution.

C.3 EMPIRICAL DIAGNOSTICS OF THE RANGE LAW

  • The pooled Figure 4 diagnostic compares group range relative to paired Hadamard with √1 −f across activation, V-cache, and extended-affine observations; the fitted intercept and slope are not used in Equation (34).The fit is descriptive and pooled rather than a separate regression for each family; its R2 is not an accuracy score for the unfitted predictor.Source: §C.3 · pp. 28–29
  • The unchanged dense-reference line tracks binned medians for 5,342 held-out Qwen3-30B-A3B experts, but scatter is wider for experts with fewer than 2,048 calibration tokens.This is a transfer diagnostic, not a universal per-expert guarantee; the experts were not excluded from their own rotation calibration.Source: §C.3 · p. 29
  • Measured Hadamard step ratios for group sizes 256/128 and 128/64 differ from the finite-g Gaussian ratios by less than 0.2% at both tested sites on two Llama checkpoints.This agreement covers the group-size factor alone, not the absolute error of the complete law.Source: §C.3 · p. 29
  • At g = 128 and k = max, median absolute relative error is 1–4% at q/k/v, 9% on both Llama down-projection inputs, and 18% on Qwen3-4B-Base down-projection input.The down-projection step is systematically smaller than predicted because removing leading heavy-tailed directions shrinks extremes more than the residual-amplitude assumption allows; the measured κPQ/κH is 0.98–1.03 at q/k/v and 0.87–0.92 at down-projection inputs.Source: §C.3 · p. 29, Table 12 · p. 29, Figure 5 · p. 30
  • The predicted ordering of six bit-accounted configurations matches measured perplexity ordering on each Llama checkpoint, but this is not an absolute-accuracy measure.The group-size ratios agree to 0.2%, and the predictor is used to order configurations and as a conservative estimate at down-projection inputs.Source: §C.3 · p. 29

D EFFICIENT DEPLOYMENT AND END-TO-END PERFORMANCE

  • On Llama-3.1-8B, the optimized implementation achieves 1.51× prefill throughput and 1.22× Graph decoding speedup over matched FP16, with 56.34% lower Graph decoding peak memory and 2.35% additional latency over matched QuaRot–Hadamard.The reported performance separates quantization-design choices, local kernel improvements, and whole-model measurements.Source: D EFFICIENT DEPLOYMENT AND END-TO-END… · pp. 29–30

D.1 DESIGN TRADE-OFFS AND THE LOW-RANK OPERATING POINT

  • The deployment operating point is k = 8: it recovers most of the Llama Hadamard-to-bf16 perplexity gap, while larger ranks give model-dependent rather than monotone gains.Alignment improves perplexity at fixed activation budget; k = 8 is not claimed to maximize accuracy for every model.Source: §D.1 · p. 30

D.2 COMPACT KERNELS AND RUNTIME INTEGRATION

  • Online rotation replaces only the R4 transform before the MLP down-projection quantizer, using two kernels and O(Tdk) projection and correction work without materializing a dense d×d rotation.The implementation reuses the INT4 matrix multiplication; the kernels communicate through a compact FP32 partial buffer before the shared quantizer and GEMM.Source: §D.2 · p. 30, Figure 6 · p. 31, Figure 7 · p. 32

D.3 BENCHMARK PROTOCOL

The deployment benchmark compares PrismQuant and Hadamard within a shared integer pipeline against matched FP16 performance references. Its timing and memory results are specific to the stated hardware, execution modes, and random-weight models.

Benchmark setup

  • The benchmark uses Llama-3.2-3B and Llama-3.1-8B on one NVIDIA A40 with a QuaRot-derived integer pipeline and random weights, making it a kernel-swap performance benchmark rather than a model-quality evaluation.The two quantized methods share packed INT4 weights, quantizer, GEMM, attention, and KV-cache implementations; FP16 is the floating-point reference.Source: §D.3 · pp. 32–33

Timing protocol

  • Prefill uses 2,048 input tokens per sequence at batch sizes 1 and 16; decode uses batch size 1 after a 2,048-token prefix for 128 causal steps, excluding the first eight from steady-state timing.Each condition has 10 warmups and 50 measured runs in each of three balanced sessions; tables report pooled timing medians.Source: §D.3 · p. 33

Graph and memory protocol

  • Graph timing excludes capture and compilation but includes replay metadata updates; peak allocated memory is measured in separate processes and reported in decimal GB.Graph paths use a preallocated 2,176-token page, and all six matched paths pass growing-cache replay checks.Source: §D.3 · p. 33

D.3.1 END-TO-END GAINS AND MEMORY EFFICIENCY

The end-to-end study reports prefill, decode, and memory outcomes against matched baselines, then distinguishes local frontend and kernel measurements from whole-model performance. Its deployment evidence is limited to a shared packed-INT4 pipeline and does not provide a checkpoint-level joint accuracy–speed measurement.

D.3.1 END-TO-END GAINS AND MEMORY EFFICIENCY

  • With both methods using Graph replay, PrismQuant incurs 1.96% additional latency on 3B and 2.35% on 8B relative to Hadamard.On 3B, its 16.65 ms/token Graph result remains above the FP16 reference of 15.04 ms/token.Source: §D.3.1 · p. 33
  • On Llama-3.1-8B, PrismQuant achieves 1.22× faster Graph decoding than matched FP16 and reduces peak allocated memory from 18.77 to 8.19 GB, a 56.34% saving.Across the four prefill conditions, throughput remains within 1.50% of Hadamard; PrismQuant reaches 1.25–1.51× matched FP16 throughput.Source: §D.3.1 · p. 33

D.4 KERNEL-LEVEL EVIDENCE AND MEASUREMENT SCOPE

  • The optimized rotation-plus-quantization frontend is faster than QuaRot at all three tested 8B shapes, with paired speedups of 1.12–1.73×; on 3B it is 1.25× faster for single-token input but costs more at larger token batches.These are local frontend measurements, not end-to-end model speedups.Source: §D.4 · p. 34, Table 14 · p. 34
  • At T = 1, pre-bound launches reduce local elapsed-time ratios to 0.53 on 3B and 0.63 on 8B; at T = 2048, the Tensor Core Kernel B path reduces complete R4 path ratios to 0.38 and 0.33.The latter ratios compare full transform paths, not Kernel B alone.Source: §D.4 · p. 34
  • Deployment establishes compact-transform efficiency within a shared packed-INT4 pipeline, but not checkpoint-level joint accuracy–speed performance.The performance backend uses per-token symmetric activation quantization and KV interfaces distinct from the native group-128 asymmetric accuracy protocol; recorded large-accumulator FP16-conversion failures remain a numerical limitation.Source: §D.4 · p. 34

E LOCAL ACTIVATION VISUALIZATIONS

Figures 8–10 visualize local activation magnitudes at query- and down-projection inputs on Qwen3-8B-Base. They distinguish end-to-end activation geometry from the local effect of rotation.

Visualization setup

End-to-end query-projection inputs

  • Figure 8 shows residual-stream activations entering q proj before input quantization; rotated rows include upstream quantization effects, unlike the unrotated reference.The view describes activation geometry along the full quantized forward path.Source: E LOCAL ACTIVATION VISUALIZATIONS · p. 35, Figure 8 · p. 36

End-to-end down-projection inputs

  • Figure 9 examines gated MLP activations after online R4 and before down-projection quantization, where PrismQuant aligns directions; large within-group components need not widen the group range.The raw surfaces indicate energy redistribution, not a direct ranking of quantization quality by peak height.Source: E LOCAL ACTIVATION VISUALIZATIONS · p. 35, Figure 9 · p. 37

Paired-local query-projection inputs

  • Figure 10 applies each transform to identical captured q proj inputs, isolating local rotation effects; changed local peak heights need not imply changed total activation energy.PrismQuant uses its deployed global R1 basis rather than a separately fitted rotation for each displayed block.Source: E LOCAL ACTIVATION VISUALIZATIONS · p. 35, Figure 10 · p. 38
Loading 2609.32429v1…