Source-linked AI summary
StagQ: Constraint-Driven Multi-Precision Weight Quantization for LLMs
Zhe Wei, Mengqi Guo, Yuan Yuan, Jiunn Bin Lim, Boyi Pan, Michael Bi Mi
TL;DR
StagQ targets fleet serving, where conventional PTQ requires separate checkpoints for different precisions. It combines a prefix-readable weight stream with a sparse record and leads the strongest multi-precision baseline at two bits across three models.
Problem
Fleet deployments need several precision operating points, while conventional PTQ requires each precision to be quantized, validated, and stored as a separate checkpoint.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
The single-checkpoint approach trades separate precision-specific checkpoints for a sparse side-record cost on every read.
Source: §1 · p. 1, §3.6 · p. 6On Llama-3.1-8B, StagQ-XS spends 0.13 bits per weight on held-out entries, and that overhead persists at every read.
Source: §3.6 · p. 6
Abstract
from arXiv · showhide
Serving a large language model (LLM) across a fleet of deployments requires several weight-precision operating points. Multi-precision formats serve them all from one stream whose prefixes are valid lower-precision codes, instead of storing multiple copies. We present StagQ, a multi-precision weight format whose main stream is a 2-bit group-wise affine base followed by a configurable number of 1-bit refinement planes on a dyadic step schedule. Every supported precision is a readable prefix, decoded by an affine map derived from metadata shared across all precisions, with no per-weight lookup. A sparse side record, filled both before and after the grid is fitted, holds out the few weights the grid serves worst. We report two configurations of the encoder. At two bits the cheaper one leads the strongest multi-precision baseline on Llama-3.1-8B, Phi-4, and OLMo-2-7B by 3.1 to 7.0 MMLU points, at a slightly lower logical rate. At three bits it leads on Llama-3.1-8B, leads on Phi-4 at a higher rate, and ties on OLMo-2-7B. At four bits it ties on all three, at a higher rate. In a batch-one matrix-vector product on an NVIDIA A100 GPU, timed on synthetic weights, our kernel is faster than the two baseline kernels in most shape-precision cases.
1 Introduction
StagQ addresses fleet deployments that need multiple weight precisions without maintaining separate checkpoints. It combines a prefix-readable affine stream with sparse held-out weights and reports benchmark and kernel comparisons.
1 Introduction
- Fleet deployments vary in capacity, concurrency, latency targets, and request priority, while conventional PTQ stores each precision as a separately quantized, validated, and stored checkpoint.A multi-precision format instead stores weights in one stream whose every prefix is a valid lower-precision representation.Source: §1 · p. 1
- The main stream pairs a group-wise affine 2-bit base with signed binary refinement planes on a dyadic schedule, decoded by one shared affine map without per-weight lookup.The base and plane fields read at each supported precision concatenate into one integer, and plane ordering lets lower-precision loads fetch fewer bytes.Source: §1 · p. 2
- Two guards hold out extreme weights in a sparse side record, first before fitting the grid and again after it is fixed.In multi-precision quantization, the held-out set affects what every precision reads, so choosing those weights is part of designing the base grid.Source: §1 · p. 2
- At two bits, StagQ leads the strongest multi-precision baseline on MMLU for Llama-3.1-8B, Phi-4, and OLMo-2-7B.At three bits it leads on Llama-3.1-8B, leads on Phi-4 while spending more bits, and ties on OLMo-2-7B; its A100 batch-one kernel is faster than both baseline kernels in most shape–precision cases on synthetic weights.Source: §1 · p. 2
2 Background
Multi-precision quantization makes successive prefixes valid representations of the same weights, but that constraint limits the grids available at matched rate. StagQ fits one shared affine pair across precisions and reads refinement planes by prefix.
2.1 Multi-Precision Quantization
- A prefix-readable format has ordered distortions and cannot beat the best fixed-precision quantizer at matched rate under the same rate, distortion, calibration, and decoder conditions.Successive refinability is equality in this ordering at the rate–distortion limit, which holds only for particular sources and distortion measures.Source: §2.1 · p. 2
- Recent LLM formats use slice nesting, plane or residual nesting, or lookup-based reconstruction, with different precision-specific metadata and decoder trade-offs.Any-Precision LLM refits lookup tables per output channel and precision; StagQ instead jointly fits one stored pair and reconstructs shallower reads at merged-cell midpoints.Source: §2.1 · pp. 2–3
2.2 Designing the Grid
- A multi-precision stream reconstructs each weight from a prefix of planes, with a base term, plane steps, and digit maps determining the grid.The group shares stored constants, while the serving system constrains the metadata and decoder choices.Source: §2.2 · p. 3
- At each read precision, a group-wise affine grid maps the integer formed by its fields using a step and an origin.The origin is the value assigned to code u = 0 under midpoint reconstruction; the interior code cells have width equal to the step.Source: §2.2 · p. 3
- The grid design choices correspond to the affine origin and displacement, plane steps that set spacing, and digit maps that determine the integer code.These are the same base-term, step, and digit-map choices introduced for the plane construction.Source: §2.2 · p. 3
3 The StagQ Framework
StagQ combines a prefix-readable shared grid with a sparse record for held-out weights, then fixes reconstruction points, refinement steps, and digit layout for multi-precision decoding.
3 The StagQ Framework
- StagQ pairs a two-bit base and one-bit refinement planes with a sparse side record for held-out weights that serves every precision.The evaluated instance exposes reads p2 through p6; each added plane raises payload by one bit per weight without adding metadata.Source: §3 · p. 3
3.1 Reconstruction Points from Measured Weights
- Measured cell centroids shift toward the group center through p5, and that displacement can be absorbed by rescaling the grid without per-code corrections.The measurement uses checkpoint weights because grouped LLM weights lack an established density for a closed-form centroid.Source: §3.1 · p. 4
- Fitted grids clip a few extremes to make levels finer across the group body, with no level placed on a group extreme.This pattern was observed across seven projections of four sampled Llama-3.1-8B layers.Source: §3.1 · p. 4
3.2 Step Schedules under a Metadata Budget
- StagQ derives every refinement step from the base step s using an unscaled dyadic schedule; globally rescaling those steps lowered median output error by at most 0.6%.The released format keeps the global rescale γ at 1.Source: §3.2 · pp. 4–5
3.3 Digit Maps and Layout for the Kernel
- The decoder constraints—fixed-width addressability, affine decoding without per-weight lookup, and prefix reads—lead to a two-bit base followed by signed one-bit residual refinements.Refinement signs use the affine digit map φk(b) = 2b − 1.Source: §3.3 · p. 5
3.4 The Resulting StagQ Main Stream
- Every precision decodes the main stream as one affine map of an integer, with its offset and step derived from the single stored pair (lo, s).The side record is applied outside this map, so a complete runtime must fuse it or report its traffic separately.Source: §3.4 · p. 5
3.5 Holding Out the Extremes
- On Phi-4’s 80 attention matrices at g=32, pre-grid selections remove 26.9% of two-bit error, versus 6.2% carried by post-grid selections.The pre-grid guard uses a ratio or gain criterion; the post-grid guard scores residual error, and the gain-per-bit score is heuristic because it includes refit recovery and depends on candidates in other groups.Source: §3.5 · p. 6
3.6 The Encoder
- The encoder activation-scales input channels before grouped fitting, then refinement planes encode residual signs while skipping positions already held in the side record.On Llama-3.1-8B, StagQ-XS spends 0.13 bits per weight on gain-criterion entries, and that record is carried at every read.Source: §3.6 · p. 6
4 Experimental Results
Experiments compare StagQ with multi-precision baselines on three dense models and evaluate its quantized kernels. StagQ-XXS has stronger low-bit MMLU results, while kernel outcomes vary with shape and decode-path fusions.
4.1 Experimental Settings
- Evaluation covers three dense models, quantizing transformer-block linear layers while leaving embeddings and language-model heads in bfloat16; reported rates include payload, side record, and declared metadata.StagQ-XXS runs at 2.34–2.35 bpw at two bits, within 0.04 bpw of the strongest baseline on all three models.Source: §4.1 · pp. 6–7
4.2 Accuracy Evaluation
- At 2–3 bits, StagQ-XXS scores higher MMLU than the strongest baseline on all three models; OLMo-2-7B’s 0.9-point 3-bit lead is within the tie band.XXS is cheaper in five of six model–precision cases; at 3 bits on Phi-4, it spends about a third of a bit more for a 1.2-point lead.Source: §4.2 · p. 7
- From 4 bits, StagQ-XXS ties Any-Precision LLM on MMLU despite costing 0.15–0.30 bpw more; at 5–6 bits on Llama-3.1-8B, MMLU and CSR differences count as ties.XXS has lower C4 perplexity than every baseline at 2–4 bits on Llama-3.1-8B and OLMo-2-7B, though some best baselines are cheaper.Source: §4.2 · p. 7
- On Llama-3.1-8B at g=32, both guards lower two-bit perplexity from 28.47 to 20.86, still above AnyBCQ’s 19.03; their MMLU gains shrink at higher precision.The guards add 2.3 and 1.6 MMLU points at three and four bits, versus 0.35 and 0.03 at five and six bits; every read still carries their side record.Source: §4.2 · p. 8
- On Llama-3.1-8B, dedicated single-precision fits spend slightly more rate for a 3.6-point two-bit MMLU gain; three- and four-bit differences remain within the tie band.The dedicated fit is cheaper by under 0.01 bpw at three and four bits, with MMLU differences of −0.03 and +0.72 points.Source: §4.2 · p. 8, Table 4 · p. 8
4.3 Kernel Evaluation
- On synthetic weights, the StagQ kernel is faster than Any-Precision LLM in 24 of 27 shape–precision cases and than AnyBCQ in 23.All seven losses occur on attention shapes timed with per-group guard space; omitting that space on feed-forward shapes favors StagQ.Source: §4.3 · p. 8
- With format-independent fusions, StagQ’s end-to-end decode path is faster in 14 of 18 comparisons; unfused at three bits, it is slower than the fastest baseline on all three models.All quantized decoders are faster than the fp16 decoder, with StagQ faster by a factor of 1.48 to 1.86.Source: Table 2 · p. 9, §4.3 · p. 9
5 Discussion and Conclusion
StagQ serves multiple precisions from one prefix-readable stream and uses a sparse side record for difficult weights. Its low-bit MMLU results lead or tie the strongest baseline in most reported comparisons, while larger-model and component-ablation evaluations remain future work.
5 Discussion and Conclusion
- At 2 bits XXS leads MMLU on all three models; at 3 bits it leads Llama-3.1-8B and higher-rate Phi-4, ties OLMo-2-7B, and from 4 bits ties Any-Precision LLM at higher rate.On synthetic weights, the StagQ matrix–vector kernel is faster than both baseline kernels in most shape–precision cases.Source: §5 · p. 9, Table 5 · p. 9
- One prefix-readable main stream uses a shared affine decode without per-weight lookup, while two guards hold difficult weights in a sparse side record.The Table 5 kernel comparison reports median speedup over cuBLAS on synthetic weights.Source: §5 · p. 9, Table 5 · p. 9
- Evaluation of larger and mixture-of-experts models and component-wise encoder ablations remains future work; production serving requires further kernel engineering.Source: §5 · p. 9
A Limitations
The reported evidence is limited by rate-accounting overhead, evaluation scope, non-matched baseline rates, and incomplete ablations. These constraints bound how broadly the comparisons and efficiency results can be interpreted.
Rate
- At two bits on Llama-3.1-8B, real-weight loading reads 2.47 bits per weight against 2.35 charged because feed-forward guard slots are allocated per row.Allocating slots by output row would read 2.39 bits per weight; adding an attention-group presence bit would bring the rate to 2.354, but both require variable-length record decoding.Source: A Limitations · pp. 11–12
Scope of the evidence
- Quality results cover three dense models and no mixture-of-experts model; timing evidence is batch-one on one A100 with synthetic weights.Real weights cause slightly more shared-memory conflict, and although byte counts imply 4.7× the fp16 speed, the timed decoder is only 1.6× as fast.Source: A Limitations · p. 12
Comparisons
- Every baseline comparison differs in rate, and several baseline runs depart from their authors’ released configurations or reported results.The Any-Precision LLM WikiText-2 perplexity differs from the reported value at two bits, while its three- and four-bit values agree within 0.03.Source: A Limitations · p. 12
Ablations and controls
- The encoder analysis lacks ablations of λ, the AWQ exponents, and the fitting of (lo, s), while the single-precision control only bounds prefix cost within this encoder.The control covers Llama-3.1-8B and does not bound prefix cost against the best fixed-precision quantizer.Source: A Limitations · p. 12
B Encoder Configurations
Encoder settings vary by model and configuration, with guard criteria and thresholds assigned at matrix or block level. Other fitting choices are shared across models and configurations.
Encoder settings
- Group sizes and guard criteria are set per model and configuration, while hold-out thresholds are set per block.The two pre-grid guard criteria are exclusive per matrix; Phi-4’s fused gate/up projection uses only the post-grid guard.Source: B Encoder Configurations · p. 12, Table 6 · p. 13
Shared settings
- The remaining settings are shared by all models and configurations, and in those settings the gain criterion always scores activation-weighted two-bit error.Source: B Encoder Configurations · p. 12
- Each matrix’s codebooks are fitted from an unstored initial pair that places four levels symmetrically, with a step of 0.29 of group range and outer levels 6% inward.On the sampled layers, fitted grids also remain narrower than the group range.Source: B Encoder Configurations · p. 12
C Optimality of the Dyadic Schedule
The dyadic refinement schedule is optimal under uniform conditional density and interval-splitting assumptions. The paper also tests a global step rescaling because measured centroid displacement violates one of those assumptions.
Optimality
- Under interval cells, binary interval splits, uniform conditional density, and scalar squared-error reconstruction, the dyadic schedule is optimal.The schedule applies only above the fitted four-level base; nonuniform conditional densities shift optimal thresholds and reconstructions away from midpoints.Source: C Optimality of the Dyadic Schedule · p. 13
- For a uniform parent cell, the minimum-error binary split is at its midpoint, with each child reconstructed by its centroid.Because either child remains uniform, recursion gives plane steps t_k = s/2^(k−1) and precision-M cell width s_M = s/2^(M−2).Source: C Optimality of the Dyadic Schedule · p. 13
Testing the schedule against the displacement
- The schedule’s uniform-density condition fails for measured centroid displacement, motivating a format-wide refinement-step factor γ that adds no per-group metadata.The test measures precision-M output error under calibration activations from the same pool as the base fit, with the pre-grid guard ratio criterion set to T=4.Source: C Optimality of the Dyadic Schedule · p. 13
- Across 28 Llama-3.1-8B matrices, the best γ differs by group size, while rescaling lowers median error by at most 0.6%; the released format keeps γ = 1.At g = 16 the median is lowest at γ = 1; at g = 32, γ = 1.05 lowers it by 0.6%, and γ = 1.1 is within 0.16% of that result.Source: C Optimality of the Dyadic Schedule · p. 14
D The Centroid Shape on Further Checkpoints
Measurements on three further dense checkpoints reproduce the center-proportional centroid displacement at p3–p5, but the pattern weakens at p6 on three of the four checkpoints.
The centroid shape on further checkpoints
- At p3–p5, the proportional fit is reproduced across the further checkpoints, and its crossing sits at the group center across all four checkpoints.Across four checkpoints, slopes span −0.73 to −0.70 × 10^-2 δu per code step at g = 16 and −0.97 to −0.94 at g = 32; R2 ranges from 0.980 to 0.999.Source: D The Centroid Shape on Further… · p. 14
- At p6, the curve falls below 1/2 on three checkpoints and its crossing shifts up to 0.16 of the half-width off center.On these three, R2 is 0.81–0.83 at g = 16 and 0.87–0.89 at g = 32.Source: D The Centroid Shape on Further… · pp. 14–15
- OLMo-2-7B is the p6 exception, retaining R2 = 0.994 at g = 16 and 0.997 at g = 32.The other three checkpoints show the off-center crossing and lower fit values.Source: D The Centroid Shape on Further… · p. 15
E Encoder Stages
The encoder calibrates and scales weights, selects a base codebook, and uses pre- and post-grid guards with a sparse side record. Group-size configurations differ in their three-bit MMLU gains.
Scale
- Calibration uses one pool of 200 fixed 2,048-token C4 windows, and AWQ-scales the weight matrix using input-channel scales.The inverse scale folds into a preceding normalization when one exists; otherwise, the timed decoder applies it at runtime on Llama-3.1-8B and Phi-4, but omits it on OLMo-2-7B.Source: E Encoder Stages · p. 15
Base
- The base codebooks use one of three placement rules, selected per matrix by the first term of Equation (5); base codes use nearest rounding without GPTQ error propagation or feedback.A matrix falls back to uniform placement if its six-bit error exceeds one tenth of its two-bit error under the selected codebooks.Source: E Encoder Stages · p. 15
Guards
- The post-grid guard selects positions using activation-weighted two-bit error, either taking the largest per output row or ranking error per stored bit and thresholding.The gain-criterion measurements of 26.9% and 6.2% apply to Phi-4’s 80 attention matrices at g=32 and τ=10^-8.Source: E Encoder Stages · p. 15
Side-record storage
- Side-record entries use different encodings by guard criterion: fp16 values with flat indices under the ratio criterion, and fixed-point codes with per-row scales under the gain criterion.Post-grid entries store fp16 residuals against the two-bit base; gain-criterion rows also carry a two-bit count of post-grid entries.Source: E Encoder Stages · p. 15
Group sizes
- At three bits, StagQ-XXS gains 1.59 MMLU points over its g=32-throughout starting configuration, while StagQ-XS gains 0.61 over g=16-throughout, within the tie band.From four bits up, neither configuration differs from its starting configuration by more than 0.42 MMLU or CSR points.Source: E Encoder Stages · p. 15
F Per-Shape GEMV Latency
On attention shapes, the StagQ kernel loses most comparisons at two bits but wins most at three and four bits.
Per-shape results
- On attention shapes, the StagQ kernel loses five of six comparisons at two bits and wins ten of twelve at three and four bits.The timed layout reserves an extra half bit per weight for attention guard space, but none for feed-forward shapes, favoring StagQ in feed-forward timings despite real matrices holding guard entries.Source: F Per-Shape GEMV Latency · p. 16
Timing interpretation
- Table 5 speedups can differ slightly from ratios recomputed from Table 8 because Table 8 uses median cuBLAS timings at each width.Table 8 reports the median of three timings per shape, one at each bit width.Source: F Per-Shape GEMV Latency · p. 16
G Kernel Implementation
The implementation decodes packed bit planes with a CUDA C++ GEMV kernel, while the accompanying bit-plane-coder discussion explains how unread planes determine a reconstruction cell and its placement.
G Kernel Implementation
- The CUDA C++ GEMV kernel keeps the AnyBCQ tiling, half-precision accumulation, and atomic-add structure, so timing differences come from the format rather than surrounding code.It reads contiguous M-bit digits repacked at load, uses constant memory for the codebook-pair table in the timed kernel, and applies both guard stages in one pass without branching.Source: G Kernel Implementation · p. 16
- The decoder’s table maps each codebook pair to (lo, s), and is loaded once per group of g weights in the described decode path.The timed kernel reads the table from constant memory instead.Source: G Kernel Implementation · p. 16
H The Grid of a Bit-Plane Image Coder
- In the bit-plane coder, reading M magnitude bits reconstructs a coefficient within the cell left open by unread planes; the base term places it at fraction δ of the cell width.Equation (1) is matched by binary digit maps and sign- and step-scaled plane terms.Source: H The Grid of a Bit-Plane Image Coder · pp. 16–17
- Under the Laplacian model for wavelet detail coefficients, squared-error-minimizing reconstruction lies at a fixed fraction of each cell; Pearlman & Said (2008) report δ ≈0.38 works well in practice.In this grid, δ is the sole design choice.Source: H The Grid of a Bit-Plane Image Coder · p. 17