PaperlayerView exact revision on arXiv

Source-linked AI summary

ApexQuant: Data-Free Elastic Quantization by Residual Re-Isotropization

Aksel Fristrup, Sumit Pandey, Ankit Kariryaa

arXiv:2610.07904v1cs.LG

TL;DR

Low-bit calibration-free quantization has trailed calibration-dependent methods, even though legal and privacy constraints can make calibration corpora unavailable. ApexQuant recursively quantizes rotated residuals so each prefix supplies a valid rate, with the decay law predicted before reading weights. Across matched bit budgets, it outperforms calibration-free baselines and narrows the gap to calibrated methods, while calibrated trellis remains stronger at both rates.

  • Problem

    Low-bit calibration-free quantizers have trailed calibrated methods, although licensing and privacy constraints can make representative calibration data unavailable.

    Source: §1 · p. 1

  • Method

    ApexQuant repeatedly rotates and quantizes residuals with a data-free codebook, yielding a closed-form decay law and valid lower-rate prefixes.

    Source: §1 · pp. 2–3

  • Results

    At two bits, ApexQuant has the lowest KL among five data-free baselines on all four language models; at four bits, it is lowest among methods stored at its rate, but trails calibrated trellis at both rates.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Because it needs only weight access, ApexQuant can be applied to Earth-observation and medical imaging without modification.

    Source: §1 · p. 3, §7 · p. 9

    ApexQuant treats directions equally, and on LLaMA2-7B calibrated QTIP reaches perplexity 5.17 versus 5.61 at four bits and 5.86 versus 71.80 at two bits; frontier-scale full-forward evaluation remains unverified.

    Source: §6 · p. 9

Abstract

from arXiv · show

We introduce ApexQuant, a calibration-free quantization method that recursively re-quantizes the residual error, serving as a refinement layer on top of existing quantizers. We establish that a fresh random rotation returns each residual to the uniform distribution on the hypersphere, which characterizes the rate of progressive error decay across successive passes. This result lets us determine, before any weight is read, how many passes a layer needs for a target weight-space error. Every prefix is itself a valid lower-rate model, so one artifact serves several precisions. We instantiate ApexQuant with three interchangeable stages, scalar, $E_8$ and trellis, and validate it on four open-weight LLMs and on Earth-observation and medical domains where in-distribution data is often unattainable as imagery arrives under restrictive licences or due to patient material under privacy constraints. Progressive re-isotropization comes within a few percent of full precision at four bits and gives the best two-bit arm we measure, in a completely data-free setting.

1 INTRODUCTION

Weight-only post-training quantization reduces inference resource demands, but low-bit calibration-free methods have lagged calibrated methods despite calibration data often being legally or practically unavailable. ApexQuant combines data-free residual refinement with nested precisions and reports strong comparisons against data-free baselines.

1 INTRODUCTION

  • Calibration-free quantization avoids calibration-data bias and remains usable when representative corpora are unavailable because of licensing or privacy constraints.The introduction cites restricted Earth-observation imagery and clinical data confined to collecting institutions.Source: §1 · p. 1
  • For a mixed-precision Mistral block stack, nested storage uses 6.09 rather than 10.16 bits per weight and decodes 1.56× faster than separate artifacts.The configuration assigns tiers (3, 3, 2, 2) across blocks, and mixed batches read each shared pass once.Source: Figure 1 · p. 2, §1 · p. 3
  • ApexQuant recursively quantizes rotated residuals with the same data-free codebook, making each pass a refinement and every prefix a valid lower-rate model.Its scalar, E8, and trellis stages share this progressive design.Source: §1 · p. 2
  • A fresh Haar rotation restores each residual's spherical marginal, and the resulting decay prediction agrees with pretrained-weight measurements to three decimal places.The introduction says the same constant generalizes to non-language domains without refitting.Source: §1 · p. 3
  • At two bits, ApexQuant achieves the lowest KL on all four language models among five published data-free baselines; at four bits, it is lowest among methods stored at its rate.Against published elastic methods it leads at four bits, while a calibrated trellis leads at both rates.Source: §1 · p. 3

2 BACKGROUND AND RELATED WORK

Related work spans calibration-dependent quantizers, data-free methods, rotation-based codebooks, and elastic precision schemes. ApexQuant differs from trained residual codecs by redrawing its rotation independently at each pass without fitting to data.

2 BACKGROUND AND RELATED WORK

  • Under a Haar-random rotation, transformed vectors are uniform on the sphere, giving coordinate marginals known in closed form for data-free codebook design.Rotation-based methods also use randomized Hadamard transforms to suppress outliers.Source: §2 · p. 4
  • QuIP# uses fixed E8 blocks and QTIP uses trellis codes, but both serve one bit-width per model.Vector quantization encodes groups jointly to reduce distortion at a given rate.Source: §2 · p. 4
  • HQQ and SINQ are structurally distinct data-free approaches that quantize unrotated weights, relying on heavy-tailed marginals or variance heterogeneity that rotation removes [Badri & Shaji, 2023; Müller et al., 2025].HQQ uses groupwise affine fitting, while SINQ derives per-column scales from row and column variances.Source: §2 · p. 4
  • Several elastic-precision methods support selectable rates, but Any-Precision LLM, MatQuant, Drop-by-Drop, MoBiQuant, and AnyBCQ require calibration or training data [Babaoglu et al., 2026; Wang et al., 2026].Their mechanisms include leading-bit models, trained integer weights, additive codebooks, a token router, and residual bit-planes.Source: §2 · p. 4
  • SoundStream and EnCodec progressively quantize residuals, but train their codebooks on compressed data and retain one fitted rotation, unlike ApexQuant's fresh data-free rotation at every pass [Zeghidour et al., 2022; Défossez et al., 2023].The passage connects these codecs to multi-stage vector quantization.Source: §2 · pp. 4–5

3 RE-ISOTROPIZATION AND THE PROGRESSIVE DECAY LAW

Fresh Haar rotations restore each residual’s direction to the uniform spherical distribution, allowing the same data-free codebook to be reused at every pass. The resulting decay and rate-loss laws quantify how many passes meet a target error and the cost of progressive refinement.

3 RE-ISOTROPIZATION AND THE PROGRESSIVE DECAY LAW

  • A fresh independent Haar rotation makes each residual direction uniform on the sphere, so the same optimal codebook remains optimal at every pass without refitting.The result applies regardless of the residual’s distribution; without a fresh rotation, decay can stall.Source: §3 · p. 5
  • The deployed randomized Hadamard transform departs from the Haar prediction by at most 1.6% on individual layers at the first pass and less than three parts in ten thousand later.Using a dense random-orthogonal rotation changes end-to-end error by at most two parts in a thousand at every pass count.Source: §3 · p. 5
  • The codebook optimum is invariant within a chosen class, but the theory does not rank scalar, E8, and trellis classes against one another.Its distortion is squared error on the normalized residual direction, a weight-space proxy rather than end-to-end quality.Source: §3 · p. 5
  • The pass rule P = ⌈ln ε/ ln ρ⌉ predicts the passes needed for target error ε before any weights are read, and the law extends to block and trellis quantizers.The analysis uses stage distortion and applies to block quantizers through the marginal distribution of the corresponding coordinates.Source: §3 · pp. 5–6
  • For a suboptimal stage, each pass incurs exactly ∆1 additional bits; across P passes the greedy penalty is exactly P∆1 relative to an optimal code, accumulating additively rather than multiplicatively.Source: §3 · p. 6

4 APEXQUANT

ApexQuant adds residual-quantization passes to a data-free stage quantizer, with the decay law determining its error curve. Its scalar, E8, and trellis stages support nested decoding, so one stored model can provide multiple precisions.

4 APEXQUANT

  • ApexQuant is a refinement layer whose error curve is determined by the distortion of whichever data-free stage quantizer is used.The three stages are arranged in increasing strength.Source: §4 · p. 7

4.1 THREE DATA-FREE STAGES

  • The scalar stage rounds coordinates with a Lloyd–Max codebook and has ρ2 = 0.1171 at two bits and d = 512.Its codebook is determined by the coordinate density fd and requires no optimization.Source: §4.1 · p. 7
  • The E8 stage quantizes eight coordinates jointly and has ρ2 = 0.0906 on rotated unit vectors.It uses the compact QuIP# E8 codebook.Source: §4.1 · p. 7
  • The trellis stage quantizes T coordinates with lookup-free computed codes and has ρ2 = 0.0720 at L = 14 on rotated unit vectors.Its minimum-error walk is found by the Viterbi algorithm in O(2^L T) time.Source: §4.1 · p. 7

4.2 ELASTIC INFERENCE

  • A model encoded with P passes can be read at any prefix as the bit-identical lower-pass artifact, allowing one file to serve several precisions and be upgraded by appending indices.For mixed-tier batches, the weight traffic equals that of the deepest tier alone.Source: §4.2 · pp. 7–8

5 EXPERIMENTS

Across language and non-language evaluations, ApexQuant’s measured error follows its predicted geometric decay, while nested passes support mixed-precision deployment. Evaluation also shows tiered preservation on Earth-observation and pathology encoders.

5 EXPERIMENTS

  • The evaluation covers four language models on WikiText-2 and Clay v1.5 and UNI on EuroSAT and PatchCamelyon, using KL against each unquantized model as the primary metric.The non-language models are evaluated as frozen encoders with linear probes.Source: §5 · p. 8

5.1 THE PROGRESSIVE DECAY LAW HOLDS ON REAL WEIGHTS

  • At b = 2, measured relative errors 0.3419, 0.1170, 0.0400, 0.0137 track the weight-free prediction to at least three decimal places at every pass.At b = 4, measured errors are 0.0973, 0.0095, 0.0009, 0.0001 against 0.0972^P; per-pass ratios remain 0.342 and 0.097.Source: §5.1 · pp. 8–9
  • With a fixed rotation, per-pass ratios climb 0.418, 0.597, 0.734 and decay stalls; dense random-orthogonal substitution changes error by at most two parts in a thousand.Perplexity shows the same ordering but by smaller margins than KL.Source: §5.1 · p. 9

5.2 ELASTICITY IN OPERATION

  • On Mistral-7B, nested mixed-precision serving takes 13.95 ms versus 21.76 ms and 6.09 versus 10.16 bits per weight versus independent artifacts.The latency advantage requires a deep batch; storage reduction holds unconditionally.Source: §5.2 · p. 9

5.3 BEYOND LANGUAGE

  • Across the three tiers, KL falls from 0.641 to 0.033 to 0.003 on UNI and from 0.224 to 0.010 to 0.002 on Clay.The tiers are 2.06, 4.13 and 6.19 bits; the table caption notes that only KL separates Clay’s tiers.Source: §5.3 · p. 9, Table 4 · p. 9

6 LIMITATIONS

ApexQuant’s data-free design treats weight directions equally, and its theory and evaluation have specific scope limits.

6 LIMITATIONS

  • At four bits on LLaMA2-7B, QTIP reaches 5.17 perplexity versus ApexQuant’s 5.61; calibration can steer rounding away from output-sensitive perturbations.At two bits the reported perplexities are 5.86 versus 71.80.Source: §6 · p. 9
  • Proposition 1 assumes Haar rotations, whereas deployed Hadamard rotations do not satisfy its hypotheses; the measured gap under two parts per thousand is not a bound.The largest end-to-end evaluation is 7B parameters, so the decay law has not been verified on a full forward pass at frontier scale.Source: §6 · p. 9

7 CONCLUSION

ApexQuant recursively re-quantizes residuals, using fresh rotations to preserve the same spherical marginal and a closed-form error curve. Its conclusion also reports fast implementations and performance against calibration-free baselines, while the appendix details checked proofs.

7 CONCLUSION

  • ApexQuant recursively re-quantizes residuals; fresh Haar rotations restore the same spherical marginal, yielding a closed-form error curve and valid lower-rate prefixes.The method’s scalar and E8 stages are near-instant, and its Triton Viterbi encoder quantizes models up to 68 times faster than the baseline.Source: §7 · pp. 9–10
  • The resulting models outperform calibration-free baselines at matched bit budgets and substantially narrow the gap to calibrated counterparts.The conclusion also reports quantization up to 68 times faster than the baseline with the Triton Viterbi encoder.Source: §7 · p. 10

A PROOFS

  • The Lean 4 development checks 154 public theorems without introducing axioms or using sorry; Proposition 1 is formalized for Haar measure on O(d).Its proof establishes that the uniform law is the only rotation-invariant probability measure on the sphere.Source: A PROOFS · p. 13
  • Corollary 1 establishes optimality within a codebook class, including residuals after P passes, but does not compare codebooks across classes.Per-block nearest rounding is checked for product codebooks, covering coordinate-wise Lloyd–Max and per-block E8P instances.Source: A PROOFS · p. 14
  • Proposition 2 derives geometric decay from independent rotations and weights, a shared right-invariant rotation law, sphere transitivity, and no exact pass.The proof identifies ρ with codebook stage distortion and shows the hypotheses can be satisfied on O(d) for every d ≥ 1.Source: A PROOFS · p. 14

B HAAR IN THEORY, HADAMARD IN PRACTICE

The deployed randomized Hadamard transform is a finite rotation family, so it does not meet Proposition 1’s Haar-rotation hypotheses. Measurements compare its reconstruction error with dense random-orthogonal rotations.

B HAAR IN THEORY, HADAMARD IN PRACTICE

  • Replacing Hadamard with dense random-orthogonal rotations changes end-to-end error by at most two parts in a thousand, without a growing gap across passes.The measurement covers eight Clay v1.5 encoder layers at d = 512 and Phi-1.5 decoder linears at both deployed group sizes.Source: B HAAR IN THEORY, HADAMARD IN PRACTICE · p. 15
  • At d = 256, the substitution gap grows from 1.4 to 2.0 parts in a thousand with pass count, while remaining at or below two parts in a thousand.Hadamard per-pass values match between Phi-1.5 and Clay to all but the fourth digit at the third pass.Source: B HAAR IN THEORY, HADAMARD IN PRACTICE · p. 15
  • Each pass shares its fresh sign pattern across groups, preserving unbiased per-pass rate but reducing layer-level concentration; the model-level average tracks ρ^P to three decimals.The reported concentration qualification concerns layer-level behavior.Source: B HAAR IN THEORY, HADAMARD IN PRACTICE · p. 15

C KERNEL IMPLEMENTATION

The trellis encoder uses Viterbi search to find a minimum-distortion walk, with computation parallelized across states and path costs kept in registers. Register capacity limits the accelerated width, so the shipped configuration uses L = 14.

C KERNEL IMPLEMENTATION

  • The Triton Viterbi encoder finds a minimum-distortion trellis walk by selecting each state’s best incoming path, with a serial time recursion that is parallel across states.The kernel keeps current path costs in registers and implements the add-compare-select sweep with register permutations and pairwise minima; separate-kernel traceback accounts for approximately 4% of encoding time.Source: C KERNEL IMPLEMENTATION · p. 15
  • The register-resident kernel is limited to 2L ≤ 214; at L = 16 it spills and falls back to a reference implementation, so L = 14 is shipped.At L = 16 the fallback reaches 2.5× rather than 68×; the wider width has lower ρ2 but cannot use the register-resident kernel.Source: C KERNEL IMPLEMENTATION · pp. 15–16, Table 5 · p. 16
  • The three interchangeable stages share the same normalized rotated-residual input and output format, differing in how many coordinates they quantize jointly.Jointly quantizing 256 coordinates with Viterbi search moves ρ2 from 0.1171 for scalar rounding to 0.0720.Source: Figure 4 · p. 16
  • The benchmark compares the Triton kernel with a DRAM-storing reference on identical weights and distortion, since different tie-breaking can yield different symbols at equal path metric.At L = 12 the implementations differ by approximately one symbol per 3000 while ρ2 agrees to six decimal places.Source: C KERNEL IMPLEMENTATION · p. 16

D LANGUAGE-MODEL BASELINES: EXTENDED RESULTS

The extended language-model results compare perplexity, KL, and downstream accuracy, while the later sections specify evaluation protocols and test whether stage ordering carries through to model outputs. Stage distinctions diminish as the bit budget grows.

D LANGUAGE-MODEL BASELINES: EXTENDED RESULTS

  • At four bits, evaluated arms span 8% in perplexity and a factor of 3.5 in KL; across the three ApexQuant stages the gaps are 1–3% against 83–90%.Tables 6 and 7 extend mean KL results with perplexity and the 99.9th-percentile KL; that tail percentile agrees with mean KL on the leading two arms in seven of eight columns.Source: D LANGUAGE-MODEL BASELINES: EXTENDED… · p. 16, Table 6 · p. 17, Table 7 · p. 17
  • At matched storage, two 2-bit scalar passes yield KL 0.120 on TinyLlama versus 0.074 for one 4-bit pass; the KL ratio ranges from 1.61 to 1.46 across models.The two-pass artifact is retained for its free 2-bit prefix, not as a rate–distortion improvement over one wider pass.Source: D LANGUAGE-MODEL BASELINES: EXTENDED… · p. 18
  • At four bits, all arms fall within 1.6 points of full precision on classification; at two bits, trellis retains 15.6%–84.9% of full precision’s above-chance margin.The classification results average ARC-Easy, WinoGrande, and HellaSwag; the table marks trellis as the best data-free arm at two bits.Source: D LANGUAGE-MODEL BASELINES: EXTENDED… · p. 18, Table 8 · p. 18

E BEYOND LANGUAGE: EVALUATION PROTOCOL

  • Clay and UNI are evaluated as frozen encoders with linear probes, using shared inputs and probe recipes; UNI’s 87.81% full-precision result reflects an 8,000-patch training split, not its published accuracy.The protocol also compares a refitted probe at each tier with a fixed full-precision probe applied across tiers.Source: E BEYOND LANGUAGE: EVALUATION PROTOCOL · pp. 18–19

F STAGE ORDERING END TO END

  • On Gemma-3-12B, paired tests show stage separations at lower pass counts, but at three passes only the scalar stage remains distinguishable; no stage pair separates at six bits per weight.The measured ordering is not fully monotone with ρ: at P=1 E8P falls below the L = 12 trellis by 0.348 nats, and the stage influence diminishes as the bit budget grows.Source: F STAGE ORDERING END TO END · p. 19

G COMPARISON WITH ELASTIC AND STATIC QUANTIZATION BASELINES

ApexQuant is compared with published elastic and static quantizers on LLaMA2-7B and LLaMA3-8B. Its lower-rate artifact is a prefix of its higher-rate artifact, and it is the only compared elastic method requiring no data.

G COMPARISON WITH ELASTIC AND STATIC QUANTIZATION BASELINES

  • ApexQuant achieves lower four-bit perplexity than MoBiQuant and AnyBCQ on both models, but the margins warrant caution because reproduced reference values differ from published reports.At two bits Any-Precision LLM achieves 35.21 against ApexQuant’s 71.80; Any-Precision LLM uses calibration data, whereas ApexQuant is data-free.Source: G COMPARISON WITH ELASTIC AND STATIC… · pp. 19–20
  • At four bits, ApexQuant is near SpinQuant and QuaRot on LLaMA2-7B and below QuIP# on both models, while QTIP has lower perplexity on both models; at two bits, only QTIP avoids unusable models.At four bits QTIP’s advantage is 0.44–0.49 perplexity points, and at two bits it is better by a factor of seven to twelve.Source: G COMPARISON WITH ELASTIC AND STATIC… · pp. 20–21
  • ApexQuant is the only compared elastic method that is data-free and whose lower-rate artifact is a prefix of the higher-rate artifact.For the illustrated mixed-precision tiers, one nested stream costs 6.09 bits per weight, versus 10.16 for independent fixed-width copies, a 1.67× storage factor.Source: G COMPARISON WITH ELASTIC AND STATIC… · p. 21, Figure 5 · p. 21

H NESTED DECODING IN SERVICE

Nested decoding makes each lower-rate model a bit-identical prefix of one stored stream and supports mixed-tier batches. In the measured Mistral setup, it reduces latency and storage relative to independent artifacts, while the latency advantage depends on serving conditions.

H NESTED DECODING IN SERVICE

  • A P-pass payload’s first t passes are byte-identical to a standalone t-pass quantization, verified across all 213 code and 213 scale tensors in three published model families.The stored stream can therefore be extended by appending passes without changing earlier bytes.Source: H NESTED DECODING IN SERVICE · pp. 21–22
  • In mixed-tier batches, sorting rows by tier lets each pass’s weight traffic equal that of the deepest tier alone; measured mixed-quality serving costs D/C = 0.995.Adding two standard-tier requests to a premium batch costs 0.49 ms, against roughly 9 ms for those requests to decode alone.Source: H NESTED DECODING IN SERVICE · pp. 21–22
  • The latency comparison is an internal decode proxy based on five linear shapes from one Mistral block at a single step, extrapolated across 32 blocks, not a full production-stack measurement.It excludes attention, KV cache, normalization, rotary, embedding, and sampling.Source: H NESTED DECODING IN SERVICE · p. 22
  • On the Mistral 32-block stack at tiers (3, 3, 2, 2), nesting is 1.56× faster and 1.67× smaller than storing one artifact per tier, with kernel-identical comparisons.Storage is 6.09 versus 10.16 bits per weight.Source: H NESTED DECODING IN SERVICE · p. 22, Table 11 · p. 22
  • The latency advantage requires batches deep enough that extra premium-tier passes are no longer free; the 1.67× storage factor and mixed-tier serving remain available unconditionally.The measurements cover decode, not prefill; prefill is compute-bound and was not measured.Source: H NESTED DECODING IN SERVICE · p. 23
  • Prefix speculation does not pay in the measured cycle, running at 0.67–0.76× plain decoding even with target kernels that are 1.9× faster.The passage distinguishes this unsuccessful draft reuse from the paper’s elasticity and kernel contributions.Source: H NESTED DECODING IN SERVICE · p. 23

I THE CALIBRATION PASS

The optional calibration pass measures each layer’s effect on model output, then uses that ranking to allocate quantization passes. On Clay, this improves embedding error per stored bit, but the gain does not extend to EuroSAT accuracy and language-model results vary by model.

I THE CALIBRATION PASS

  • Sensitivity is measured by quantizing each candidate layer alone and recording output displacement, requiring 97 forward passes on Clay’s 96-layer encoder.The calibration pass is the single optional step that reads data.Source: I THE CALIBRATION PASS · p. 23
  • A greedy allocator gives every layer a base pass count, then adds passes to the highest-ranked sensitive layers without refitting codebooks or changing rotations.For example, “base 2 +3 at 25%” gives three extra passes to the most sensitive quarter.Source: I THE CALIBRATION PASS · p. 23
  • 0.0406 versus 0.0534 embedding relative error Pareto-dominates uniform three-pass allocation at 1.2 fewer bits per weight.Figure 6 reports 4.99 versus 6.19 bits per weight.Source: Figure 6 · p. 24, I THE CALIBRATION PASS · p. 24
  • 96.10% versus 96.25% EuroSAT accuracy shows no improvement from allocation, despite using 4.92 versus 4.13 stored bits.The 0.15-point difference is within the test set’s 0.43-point standard error.Source: I THE CALIBRATION PASS · p. 24

Failure on language models

  • On Phi-1.5, allocation is 1.66× worse than uniform at four bits and 1.20× worse at six bits, while TinyLlama improves at both budgets.A single-evaluation validation gate rejects the four-bit Phi-1.5 case but may not detect the smaller six-bit degradation.Source: I THE CALIBRATION PASS · p. 24
  • Current calibration-pass results are restricted to the vision encoder; multi-evaluation gating and more robust sensitivity estimation remain future work.The authors leave both approaches for future work.Source: I THE CALIBRATION PASS · p. 25

Related papers

Loading 2610.07904v1…