PaperlayerView exact revision on arXiv

Source-linked AI summary

Unfolding the Leech Lattice: Fused Multi-Shell Decoding and VRAM Layouts for 2-Bit LLM Weights

Pier-Jean Malandrino

arXiv:2609.02652v1cs.LG

TL;DR

Leech-lattice quantization has the strongest reported 2-bit quality under its own evaluation protocol, but its existing kernel decodes one shell while the rate requires a multi-shell decoder. This paper supplies a serving path for the full codebook and measures its decode-phase GEMV cost: the trellis kernel reads 2.40× fewer bytes and runs 2.27× faster than Planes14, while the reasoning-quality deficit remains unexplained.

  • Problem

    Leech-lattice vector quantization has the strongest reported 2-bit quality under its own evaluation protocol, but its existing kernel decodes one shell rather than the 301-class union required by the rate.

    Source: §1 · p. 1

  • Method

    The paper unfolds the full 301-class codebook into GPU layouts and uses a fused, divergence-free dequantize-plus-matvec kernel verified against f64.

    Source: §8 · p. 16

  • Results

    The trellis kernel reads 2.40× fewer bytes than Planes14 and runs 2.27× faster; with the output head held identical, the path gains 1.11×, 1.29× and 1.41× end to end at 4B, 8B and 14B.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The format read from VRAM, not the on-disk rate, determines which models fit on a GPU, making in-VRAM layout a distinct design axis.

    Source: §1 · p. 1, §3 · p. 5

    The MMLU deficit remains unexplained: these data cannot distinguish a 2-bit reasoning-quality loss from bias introduced by C4 calibration.

    Source: §6 · pp. 14–15

Abstract

from arXiv · show

Leech-lattice vector quantization holds the strongest reported 2-bit quality under its own evaluation protocol. Its kernel decodes one shell; we found no implementation of the multi-shell decoder the rate requires. This paper supplies one and measures its serving cost for decode-phase GEMV at batch 1. First, a serving path for the full 301-class codebook: an offline expansion into GPU layouts and a fused dequantize-plus-matvec kernel reading them without warp divergence, verified against f64. Second, the in-VRAM rate is a design axis distinct from the on-disk rate. Four bit-exact layouts timed in one process show binary bit planes beating one-hot masks on size and speed at constant bandwidth (4.80 bits per weight, 2.15x FP16). Below 4.3 bits a second, irregular stream enters; at 3.6 the decode stops being shifts and masks. Third, deployed four-bit (AWQ) and two-bit (QTIP) GEMV kernels run in the same process. The trellis kernel reads 2.40x fewer bytes than our served layout and runs 2.27x faster at near-equal fractions of their byte bounds: the time gap tracks the traffic gap, the price of unfolding a codebook too large for a lookup table. Fourth, the validity envelope: the trellis kernel outruns our no-weights control, so our launch geometry sets that floor, and on a second memory hierarchy every lattice arm falls below FP16. With the output head held identical across arms, the kernel-and-format path gains 1.11x, 1.29x and 1.41x end to end at 4B, 8B and 14B; with an int8 output head the served 4B reaches 87.0 tok/s in 2.60 GB. The quality cost, 1.38x perplexity and 14.7 MMLU points at 4B, shrinks across the three sizes measured.

1 Introduction

The paper implements and measures a serving decoder for the full multi-shell Leech-lattice codebook, addressing a gap left by a single-shell kernel. It evaluates the VRAM cost and serving performance of this path alongside deployed GEMV kernels and end-to-end models.

1 Introduction

  • The fused decoder serves all 301 classes through offline GPU-layout expansion and divergence-free dequantize-plus-matvec, verified against f64.On an L40S it reaches 65% of its byte bound, measured against the FP16 control’s bandwidth.Source: §1 · p. 2
  • At 4.80 bits per weight, Planes14 runs at 2.15× an FP16 matvec; the layout curve turns below 4.3 bits and collapses at 3.6.Four bit-exact layouts span 5.51, 4.80, 4.34 and 3.59 bits per weight; the lower-rate formats introduce an irregular exception stream or stop using shifts and masks.Source: §1 · p. 2
  • 2.40× fewer bytes and 2.27× faster: the trellis kernel outperforms Planes14 at 61% versus 65% of the byte bound.The comparison is with deployed AWQ w4g128 and QTIP GEMV kernels in one process; launch geometry is not equalized.Source: §1 · p. 2
  • The trellis kernel outruns a no-weights control, and on an A100 every lattice arm falls below FP16.The no-weights control uses the same launches and shapes, so launch geometry sets the time floor; the headline ordering and ratios are bounded to the bandwidth-bound regime.Source: §1 · p. 2
  • 1.11×, 1.29× and 1.41× end-to-end gains at 4B, 8B and 14B accompany a 1.38× perplexity cost and 14.7 MMLU-point deficit at 4B.With an int8 output head, the served 4B configuration reaches 87.0 tok/s in 2.60 GB of VRAM; the quality deficit shrinks at 8B and 14B.Source: §1 · p. 2

2 The Fused Multi-Shell Decoder

The decoder treats each 24-weight block as a 47-bit lattice index, then expands it offline into GPU-readable records rather than decoding the index inside each matvec. The resulting kernel uses a common instruction sequence across classes and is checked against an independent f64 reference.

2.1 What a Λ24 index is

  • The Λ24(12) ball contains 111 043 117 458 000 points, so each direction index takes 47 bits; its points span 301 permutation-and-sign classes.The 48-bit block also carries one gain bit selecting between two per-tensor magnitude centroids.Source: §2.1 · pp. 2–3

2.2 Unfolding at load time

  • The 47-bit index is expanded once at load time into decoded-field records, avoiding per-class branches in the matvec.Planes14 stores a 112-bit record where the index it replaces is 48 bits; Fig. 1 also compares Slot32, Planes12x and Golay70 layouts.Source: §2.2 · p. 3, Figure 1 · p. 3

2.3 The kernel

  • Each warp computes one output row; Planes14 reads a record through a four-word window and selects magnitude levels with shifts, masks and a predicated tree.The class table is read through L1; constant memory would serialize the warp’s 32 distinct addresses.Source: §2.3 · pp. 3–4

2.4 Verification

  • Every benchmark output row is checked against an independent f64 reference before timing, while transcoding is verified bit-exact across all 150 681 600 blocks.The reference checks cover 1 105 920 rows, and upstream enumeration is locked by exact combinatorial invariants.Source: §2.4 · p. 5

3 VRAM Layouts: the In-VRAM Rate as a Design Axis

The section treats the format read from VRAM as a design axis and compares bit-exact layouts and deployed GEMV kernels under a shared decode-phase benchmark. Across the tested layouts, savings depend on both memory traffic and decode and launch costs, and the reported speedups are bounded by the memory hierarchy.

3.1 Protocol

  • The layout benchmark uses ten arms in one process on an L40S, with one batch-1 token through Qwen3-4B’s 252 projections and f64 checks before timing.Seven interleaved rounds were run, with the first two discarded; rates count the bytes streamed by each kernel.Source: §3.1 · p. 5, §3 · p. 5

3 VRAM Layouts: the In-VRAM Rate as a Design Axis

  • The section compares each layout’s VRAM rate with its speedup and achieved fraction of the FP16 control’s bandwidth, treating in-VRAM rate separately from on-disk rate.Its benchmark accounts for streamed payload, exception side channels, f32 tail columns, and per-row scales.Source: §3.2 · p. 5, §3 · p. 5

3.2 The layout scale

  • Planes14 replaces Slot32’s one-hot masks with binary bit planes, reducing the rate to 4.804 bits per weight and reaching 2.15× FP16 at unchanged bandwidth.It runs 1.14× faster than Slot32; the reported bandwidths are 431 and 428 GB/s, respectively.Source: §3.2 · pp. 5–6
  • Planes12x reaches 4.342 bits per weight without quality change, but its exception stream lowers its byte-bound fraction from 65% to 54%; Golay70’s smaller records incur decode costs.Golay70 reaches 1.34× at 30% of its byte bound, while hoisting coset logic raises this to 1.78× and 40% at unchanged stored bytes; both remain below their stated speed criteria. At 3.6 bits per weight the curve collapses, and a variable-width 2.40-bit layout runs at 0.25× FP16.Source: §3.2 · p. 7

3.3 Where deployed 4-bit and 2-bit kernels sit

  • 2.40× fewer bytes and 2.27× faster: QTIP reads 0.91 GB versus Planes14’s 2.18 GB, with both kernels near their byte bounds at 61% and 65%.The authors attribute the time gap to traffic at near-equal efficiency; the QTIP kernel uses its own shipped launch geometry.Source: §3.3 · pp. 7–8
  • The no-weights control completes the same 252 launches in 2.306 ms, slightly slower than QTIP’s 2.246 ms, bounding the launch geometry for that kernel shape.The reported difference is 2.6%, against a 0.36% resolution; the authors caution that a differently shaped kernel can pass below this control.Source: §3.3 · p. 8

3.4 Attribution to the DRAM floor

  • Fusion cuts launches from 252 to 144 and recovers 11.7% of Planes14’s time; launch geometry is the largest attributed term, independent of format.The fused served path is 1.061× end to end on the 4B; the two arms read identical bytes.Source: §3.4 · p. 8

3.5 A second memory hierarchy

  • On the A100, Planes14 reaches 0.79× FP16 versus 2.15× on the L40S, and every lattice arm falls below FP16; only the ordering transfers across the tested hierarchies.The authors bound the headline ratios to the bandwidth-bound regime.Source: §3.5 · pp. 9–10

4 End-to-End Integration

The end-to-end evaluation separates kernel-and-format gains from output-head changes, and compares both with a dense path and an in-engine quantized baseline. With the output head held identical, the gains increase across the three model sizes.

4 End-to-End Integration

  • The serving setup measures greedy decode-phase GEMV at batch 1 on one L40S, with a rotation kernel before each projection and weights-only VRAM accounting.The fused Planes14 kernel replaces 252 linear projections; the dense and fused engines emit identical greedy tokens for 88 positions before a reproducible tie-break divergence at token 89.Source: §4 · p. 10
  • The dense output head takes 26.7 ms per token, twice the cost of its 36 transformer blocks, because broadcast matmul materializes a roughly 0.8 GB vocabulary-table copy.An int8 head kernel reads its 413 MB in place and reduces that phase from 25.9 to 0.6 ms; the section holds the head identical for the kernel-and-format comparison.Source: §4 · pp. 10–11
  • 2.00× is the served-path gain at 4B: 87.0 tok/s versus 43.5 tok/s, with most of the gain coming from output-head replacement.The served series is 2.00×, 2.57× and 2.55× across the three sizes; its denominator includes an output-head path whose cost changes with vocabulary projection.Source: §4 · pp. 10–11, Table 3 · p. 11
  • 1.11×, 1.29× and 1.41× are the end-to-end gains at 4B, 8B and 14B when both arms use the same f16 output head.The 14B served path required increasing the rotation kernel’s shared-memory bound; its 128 greedy tokens matched the dense arm’s.Source: §4 · p. 11, Table 3 · p. 11
  • In its own vLLM stack, official 4-bit quantization yields 2.413× against its f16 control, versus 1.11× for the same-head comparison here; the f16 controls differ.The paper reports the two within-stack ratios side by side and does not divide them.Source: §4 · p. 11

5 Evaluation

The evaluation scores the sealed quantized artifact against FP16 and compares its quality and footprint with published 2-bit results. At 4B, the 2-bit artifact has a measurable quality deficit, which narrows across the three model sizes, while its whole-model rate is below the official AWQ checkpoint at each size.

5 Evaluation

  • The quality protocol uses one L40S harness, FP16 on both sides, and matching token fingerprints; WikiText-2 uses 12 windows and MMLU uses 2,280 questions.The quantized weights are a sealed artifact calibrated on 131k C4 tokens; the source paper used roughly two orders of magnitude more tokens from DCLM-edu.Source: §5 · p. 11, Table 5 · p. 12, Table 6 · p. 12
  • At 4B, the sealed 2-bit artifact has ×1.384 perplexity and a +14.73-point MMLU loss versus FP16.The paired 95% interval for the MMLU loss is +11.98 to +17.47 points.Source: §5 · p. 12
  • Across 4B, 8B and 14B, perplexity degradation falls ×1.3845 → ×1.2201 → ×1.1894, while paired MMLU loss falls 14.73 → 10.57 → 6.85 points.The intervals and calibration draw do not support a scaling law from three models.Source: §5 · p. 13, Figure 6 · p. 13
  • Three same-configuration 4B calibration draws give perplexities of 16.74, 15.88 and 15.10, a 10.3% range of the median; absolute degradation therefore carries calibration variance.The published artifact is not another draw from that distribution because it used a contiguous prefix from a different C4 shard.Source: §5 · p. 13
  • At 8B, the 4-bit loss against FP16 is +3.07 MMLU points with a [+1.61, +4.69] interval, while the 4B and 14B intervals contain zero.The paper says this finding is specific to the 8B and implies no monotone tendency.Source: §5 · p. 13
  • Whole-model rates are 5.162, 5.322 and 5.106 b/param versus AWQ’s 5.302, 5.956 and 5.404, respectively, at 4B, 8B and 14B.The margins are 2.6%, 10.6% and 5.5%; they track the embedding share and are not monotone.Source: §5 · p. 13, Figure 6 · p. 13
  • Planes12x serves 4B at 85.0 tok/s in 2.36 GB, gives up about 2% of Planes14’s throughput, and is the most compact served configuration measured.Its greedy tokens match the dense arm through the same tie-break.Source: §5 · p. 13
  • At matched 2-bit code rate, 4B perplexity ×1.38 sits level with QTIP and LLVQ’s 0-gain-bit line (×1.37), behind its 2-gain-bit line (×1.25); at 8B, ×1.22 trails both LLVQ lines and leads QTIP (×1.24).The compared papers use their respective protocols, and the source paper reports no error bars.Source: §5 · pp. 13–14, Table 7 · p. 14

6 Limitations and Validity Envelope

The validity envelope is bounded by the evaluation path, calibration uncertainty, decode workload, and GPU memory limits. The reported speed and quality results do not establish performance at 70B or explain the 4B MMLU deficit.

Validity envelope

  • The 4B quality score uses the dense path, with served-path equivalence relying on bit-exact transcoding and row-level f64 checks; unfolding takes 122–131 s versus 186–188 s to read 8 GB of f16.The fused 4B load time includes unfolding.Source: §6 · p. 14

MMLU deficit

  • Calibration volume is bounded on perplexity only: a contamination oracle recovers 1.6%, while a 13× volume curve recovers 1.2%; calibration composition remains untested.The calibration uses 131k tokens, about 100× below the original paper, and the source paper uses DCLM-edu rather than C4.Source: §6 · p. 14
  • The MMLU deficit cannot distinguish a 2-bit reasoning-quality loss from C4 calibration bias; source-matched requantization would settle that distinction.Calibration composition is the leading untested suspect; learned per-column scales and low-rank compensation also remain untested.Source: §6 · pp. 14–15

Kernel comparison

  • A tuning pass was attempted on neither kernel, and the 2.27× trellis advantage over Planes14 is unmeasured against Marlin's batched 4-bit GEMM and other 2-bit vector-quantized systems.The 2.27× comparison is in one process; the authors caution tuning could move the number.Source: §6 · p. 15

Batching

  • At batch 1, one activation amortizes each weight byte; batched GEMM amortizes bytes across rows, and larger models change embedding share and tail-column overhead.The measurements therefore do not establish the same whole-model rate for batched workloads or larger models.Source: §6 · p. 15

Scaling

  • Three model sizes establish a direction but no scaling law; the 70B class targeted by the memory argument was not measured.The methodology could measure that regime, but the paper does not report it.Source: §6 · p. 15

Shared-memory limit

  • The Walsh–Hadamard rotation needs one f32 per down_proj input coordinate: 14B requires the 101,376-byte opt-in bound, while 32B exceeds it by 1,024 bytes.Every layout uses the same rotation; half-precision shared activations would halve the footprint, but their accumulation depth is unpriced.Source: §6 · p. 15

7 Related Work

Related work spans low-bit quantizers and their serving kernels, from lookup-decoded codebooks to vector and trellis codes. The paper positions its full Leech-lattice decoder and byte-bound measurements within those lines of work.

Extreme low-bit quantization

  • LLVQ [29] improves on QTIP [26] with a spherical GPTQ variant on the Leech lattice, but its kernel decodes one shell; this work supplies the multi-shell decoder.Lookup-decoded E8P and IQ2 codebooks hold up to 1,024 points, whereas Λ24(12) contains 1.1 × 10^14 points and must be unfolded.Source: §7 · pp. 15–16
  • SpQR [6] and SqueezeLLM [14] retain scalar grids and isolate outliers in a sparse side channel, like the paper's sparse overlay, which also adds an irregular memory stream.AQLM [7], VPTQ [17], and GPTVQ [28] have released kernels, but the authors found none serving the multi-shell Λ24 codebook and measure none of them.Source: §7 · pp. 15–16

Rotations

  • QuaRot [2] and SpinQuant [18] made incoherence rotations practical at 4 bits, but each still applies a serving-time Hadamard transform and meets the shared-memory constraint only at some widths.The paper's rotation path likewise has a shared-memory width limit.Source: §7 · p. 16

Quantized-inference kernels

  • LUT-GEMM [22] and FLUTE [11] use fast-memory lookup tables; this paper instead asks how far below 4 bits per weight in VRAM decode cost overtakes byte savings.AWQ [16] is the deployed 4-bit reference; Marlin [9] demonstrates a mature mixed-precision kernel, including batched regimes outside this paper's scope.Source: §7 · p. 16

Performance models

  • The paper's roofline argument uses the FP16 control's measured bandwidth as the byte bound and a no-weights control to price launch geometry.On an A100 the layout ordering survives but the ratios differ, so one machine's roofline does not describe the other.Source: §7 · p. 16

Serving engines

  • The integration uses candle, while the 4-bit throughput reference was measured in vLLM [15]; candle's Linear path avoids the output-head materialization associated with the engine's broadcast matmul primitive.The two throughput measurements therefore come from different serving engines.Source: §7 · p. 16

8 Conclusion

The paper implements a fused decoder for the full Leech-lattice codebook and evaluates its memory layouts and serving cost. Its measured advantages depend on launch geometry and memory hierarchy, while quality scaling and the cause of the reasoning-focused deficit remain open.

Conclusion

  • Against deployed 4-bit and 2-bit GEMV kernels in one process, the trellis kernel reads 2.40× fewer bytes and runs 2.27× faster at near-equal efficiency; codebook unfolding is paid in bytes.Source: §8 · p. 16
  • Four bit-exact layouts span 5.51 to 3.59 bits per weight; bit planes beat one-hot masks on size and speed, while the curve turns below 4.8 bits and collapses at 3.6.Below 4.3 bits a second irregular stream enters, and at 3.6 bits decoding stops being shifts and masks.Source: §8 · p. 16
  • Launch geometry holds 39% of the gap to the DRAM floor, and projection fusion recovers 11.7% of Planes14 kernel time and 1.061× end to end; on an A100 every lattice arm falls below FP16.The trellis kernel also outruns the no-weights control, so launch geometry sets that measured floor.Source: §8 · p. 16

Open questions

  • The direction of the quality curve holds on three sizes, but there is no scaling law; whether it reaches 70B and which untested lever closes the reasoning-concentrated MMLU deficit remain open.The named untested levers are calibration composition, learned column scales, and low-rank compensation.Source: §8 · p. 16

9 Availability

The authors provide source code, paper materials, and the CSVs underlying tables and figures, with build checks against the data. A 4B artifact is published; the 8B and 14B artifacts are not.

Availability

  • The source, paper, and CSVs behind every table and figure are available in the repository, and its build rejects tables that drift from their CSVs.The measurement campaign cost under $100 in rented GPU time; the 2-bit competitor's GPL v3 kernel is fetched at a pinned commit rather than redistributed.Source: §9 · p. 17

Artifacts

  • The sealed 4B artifact is published at 1.77 GB, or 1.41 GB with the int8 embedding pre-baked; the 8B and 14B artifacts are unpublished.Requantizing those sizes would be a new calibration draw, not the same artifact.Source: §9 · p. 17

Reproduction

  • On an NVIDIA GPU, the served configuration and evaluations replay from the artifact; environment variables select layout, embedding precision, and evaluation dtype.The served configuration requires the CUDA-only rotation kernel, and each evaluation binary prints a scored-token fingerprint.Source: §9 · p. 17

Use of generative AI

The manuscript discloses extensive use of generative AI in writing, coding, testing, and statistical recomputation, while assigning responsibility to the author.

Use of generative AI

  • Claude drafted and revised the manuscript text.The disclosure identifies Claude through the Claude Code CLI in 2026.Source: Use of generative AI · p. 17
  • Claude wrote the figure-generating and table-checking scripts and portions of the CUDA kernels, Rust host code, and tests.Source: Use of generative AI · p. 17
  • The author designed the study, authorized every measurement, verified each number against its source, and accepts responsibility for the entire content.Claude also recomputed statistics from the committed measurement journals.Source: Use of generative AI · p. 17

A Statistical tests on the scale curve

The appendix reports paired uncertainty analyses for perplexity and MMLU across model sizes, including evidence that limits claims about how quality gaps change with scale.

A Statistical tests on the scale curve

  • Perplexity intervals use paired Student t intervals over 12 windows, while MMLU deltas use subject-stratified paired bootstraps and exact McNemar tests over 2,280 questions.The intervals cover the evaluation corpus, not the calibration draw.Source: A Statistical tests on the scale curve · p. 19
  • All 108 window-level perplexity comparisons agree in sign, and no reported perplexity interval contains zero; two MMLU intervals contain zero.The two MMLU intervals are FP16 −4-bit at 4B and 14B.Source: A Statistical tests on the scale curve · p. 19
  • The knee excludes zero for the artifacts as built, but one 4B calibration redraw moves it from −0.10 to +0.01.A third redraw leaves the first step itself no longer separated from zero.Source: Table 11 · p. 19
  • The MMLU gap to 4 bits steps are 6.96 points (SE 1.82, p=0.0001) and 1.40 (SE 1.68, p=0.40), with the second supporting neither flattening nor a constant pace.Source: A Statistical tests on the scale curve · p. 19
  • Against AWQ, the perplexity knee is −0.0576 [−0.1011, −0.0140] (t=−2.91), while the 8B→14B step is −1.58% [−3.14, −0.004], at the interval’s edge.Neither result supports extrapolation beyond the three measured sizes.Source: A Statistical tests on the scale curve · p. 19

B Protocol, pre-registered criteria and provenance

The paper documents its benchmark protocol, pre-registered criteria, software environment, and data provenance, while noting that the shipped kernel comparison retains asymmetries.

B Protocol, pre-registered criteria and provenance

  • The ten-arm layout benchmark ran interleaved in fixed dispatch order for seven rounds, with two rounds discarded, on one L40S in one process.Speedups use medians of per-round ratios against the FP16 control.Source: B Protocol, pre-registered criteria… · p. 19
  • The runtime used candle 0.9.2, CUDA 12.4.1, and NVRTC-compiled target-SM kernels; committed CSVs regenerate figures and checked tables fail the build on drift.The GPU drivers are also specified separately for the L40S and A100 runs.Source: B Protocol, pre-registered criteria… · pp. 20–21
  • The comparison holds matrices, shapes, process, rounds, and f64 verification equal, but preserves as-shipped differences that understate the authors’ arms.The protocol also notes three uncorrected asymmetries.Source: B Protocol, pre-registered criteria… · p. 19, Table 12 · p. 20
  • Decision criteria and predictions were fixed before their measurements, and outcomes are read from the measurement journals.The first listed criterion predates the authors’ use of pre-registration proofs.Source: Table 13 · p. 20
Loading 2609.02652v1…