PaperlayerView exact revision on arXiv

Source-linked AI summary

Breaking the 1.58-bit Barrier for Ternary LLMs

Evangelos Georganas, Alexander Heinecke, Pradeep Dubey

arXiv:2609.16338v1cs.AIcs.LG

TL;DR

Ternary storage commonly assumes equiprobable symbols, despite measured zero densities reaching 51.5% in ternary LLMs. BITCOS uses a presence bitmap and compacted sign vector, and its GEMV kernels outperform state-of-the-art 2-bit kernels across tested CPUs and GPUs.

  • Problem

    Five-trit deployment packing costs 1.625 bits per weight in practice while treating the three ternary symbols as equiprobable, although zeros reach 51.5% across measured models.

    Source: §I · p. 1

  • Method

    BITCOS stores one presence bit per weight and a compacted sign bit per non-zero, for an effective rate of 2 − z bits per weight at zero density z.

    Source: §I · pp. 1–2

  • Results

    Across tested CPUs and GPUs, BITCOS GEMV outperforms the state-of-the-art 2-bit kernel, reaching 1.28× speedup on a 64-core server Emerald Rapids CPU.

    Source: §V · p. 11

  • Takeaways & Limitations

    Against the production 2-bit format, BITCOS reduces weight traffic by 1.16–1.32× on all 29 tested models.

    Source: §V · p. 11

Abstract

from arXiv · show

Ternary Large Language Models (LLM) store every weight as one of three symbols $\{-1,0,+1\}$, so the cost of a ternary model is conventionally referenced to the information-theoretic $\log_2 3 \approx 1.585$ bits per weight. The prevailing deployment format packs five ternary weights into one byte (five-trit packing), and due to the power-of-two group sizes used in practice this rounds up to $1.625$ bits per weight. This effective storage bit-width treats the three symbols $\{-1,0,+1\}$ as equiprobable. We measure the actual symbol distribution of 29 ternary LLM models and find that zeros account for up to $51.5\%$ of all weights. Motivated by this finding, we introduce BITCOS, a simple distribution-adaptive layout comprised of a dense presence bitmap plus a compacted sign vector, and costs $2 - z$ bits per weight element given a zero density $z$ in the model's weights. BITCOS stores weights more compactly than the five-trit packing in 26 of the 29 tested models, and reaches $1.485$ bits per weight on the sparsest of them. BITCOS is amenable to efficient unpacking on modern processors and GPUs, and we present optimized unpacking sequences for AVX-512, AVX2 and Intel Xe2 GPUs. Measured against production state-of-the-art ternary matrix-vector multiplication kernels, at the zero densities real-world ternary models exhibit, the realized gain with our proposed layout is up to $1.28\times$. Finally, we illustrate end-to-end LLM inference results on 5 different platforms (client and server CPUs, integrated and discrete Xe2 GPUs) where decode throughput improves by up to $1.18\times$ on CPUs and $1.27\times$ on GPUs.

I. INTRODUCTION

The paper introduces BITCOS, which adapts ternary weight storage to measured zero density, and reports lower storage rates for most tested models. It also presents optimized unpacking and multiplication kernels, with decode-throughput gains on several CPU and GPU platforms.

I. INTRODUCTION

  • In practice, five-trit packing costs 1.625 bits per weight because power-of-two blocks such as 128 require 26 payload bytes.The theoretical reference is log2 3 ≈ 1.585 bits per weight under an equiprobable-symbol assumption.Source: §I · p. 1
  • Zeros account for up to 51.5% of weights across the measured 29 state-of-the-art ternary LLM models.The paper measures seven model families.Source: §I · p. 1
  • BITCOS uses one presence bit per weight and one sign bit per non-zero, costing 2 − z bits per weight; 26 of 29 models store more compactly than five-trit packing.The five-trit comparison favors BITCOS when zero density z exceeds 0.375.Source: §I · pp. 1–2, Figure 1 · p. 1
  • Decode throughput improves by up to 1.18× on a 64-core server CPU and 1.27× on a discrete Xe2 GPU across seven checkpoints, but gains are not universal.On an 8-core, bandwidth-rich client platform, the smaller payload does not translate into performance benefits because the unpack sequence is exposed.Source: §I · p. 2
  • The paper provides optimized unpacking and matrix-multiplication instruction sequences for x86 CPUs and Intel Xe2 GPUs, plus a roofline model for assessing CPU-kernel efficacy and limitations.The evaluation covers modern server and client CPUs and integrated and discrete GPUs.Source: §I · p. 2

II. THE BITCOS LAYOUT: BITMAP + COMPACTED SIGNS

BITCOS stores a full presence bitmap and one compact sign bit per nonzero weight, so its cost falls with zero density. The section shows the layout’s storage advantage and an AVX-512 sequence for unpacking it.

A. Layout definition and storage cost

  • The layout pairs a full-length presence bitmap with a sign vector containing one bit per nonzero weight, costing 2 − z bits per weight.The sign vector is compacted to the bitmap’s population count.Source: A. Layout definition and storage cost · p. 2
  • BITCOS beats five-trit packing in 26 of 29 models when zero density exceeds 0.375 and is more efficient than 2-bit packing at all zero densities.Its cost falls linearly with zero density.Source: A. Layout definition and storage cost · pp. 2–3

B. BITCOS unpack sequence with x86 AVX-512 instructions

C. BITCOS unpack sequence with x86 AVX2 instructions

The AVX2 path reconstructs ternary weights as int8 because its target client CPUs lack AVX-512 mask registers and fp16 compute. It materializes byte masks to form each weight before integer contraction.

C. BITCOS unpack sequence with x86 AVX2 instructions

  • AVX2 reconstructs ternary weights as int8 and contracts them with int8 activations because the target client CPUs lack AVX-512 mask registers and fp16 compute.The sequence targets AVX-VNNI-INT8.Source: C. BITCOS unpack sequence with x86… · p. 3
  • The presence and sign bits become byte masks, and W_i = p_i − 2n_i yields +1, −1, or 0 for positive, negative, or absent weights.The example reconstructs the original eight-value column, with only seven compact sign bits stored.Source: C. BITCOS unpack sequence with x86… · pp. 3–4
  • AVX2 spends five instructions materializing presence byte masks, compared with one mask-move instruction on AVX-512.Source: C. BITCOS unpack sequence with x86… · p. 4

D. BITCOS unpack sequence for Intel Xe2 GPUs

The Xe2 kernel replaces pdep with a shared-local-memory lookup table that maps presence and compact-sign nibbles to four ternary weights. Its layout and cursor logic support parallel output columns while tracking variable-rate sign streams.

D. BITCOS unpack sequence for Intel Xe2 GPUs

  • For each block, the kernel aligns sign bits, decodes eight four-row groups through SLM lookups, applies the group scale, and contracts after four groups.A block-wide population count advances the sign rank across all 32 rows.Source: D. BITCOS unpack sequence for Intel… · pp. 4–5
  • Because Xe2 lacks pdep, the kernel uses a shared-local-memory lookup table indexed by four presence bits and four compact-sign bits to return four fp16 ternary codes.The table is a precomputed four-bit pdep composed with the sign-to-code mapping.Source: D. BITCOS unpack sequence for Intel… · pp. 4–5
  • Each column’s cursor advances by the number of present weights, so successive four-bit sign windows can overlap even though the bits consumed are disjoint.Per-group population counts shift the window; a per-block count updates the gather address.Source: D. BITCOS unpack sequence for Intel… · pp. 5–6, Figure 9 · p. 5
  • Sixteen SIMD lanes map to sixteen output columns, and one four-row lookup produces 4×16 weights.Each lookup fills two adjacent register rows.Source: Figure 8 · p. 5
  • The table has 256 entries of four fp16 values, totaling 2 KB; four-row groups fit the SLM table and match the two-dword VNNI2 operand layout.Source: D. BITCOS unpack sequence for Intel… · p. 6
  • A single three-word sign gather serves two 32-row blocks, while adjacent columns’ bitmap words are read together.The two memory access patterns reflect contiguous bitmap words and per-column sign cursors.Source: D. BITCOS unpack sequence for Intel… · p. 6

A. Experimental platforms

The evaluation uses three x86 CPU platforms and two Xe2 GPU platforms with differing core counts, memory types, and measured bandwidths. The CPU instruction sets determine which BITCOS kernel is run.

A. Experimental platforms

  • The three CPU platforms are a 64-core Xeon Platinum 8592+ (EMR), a 24-core Core Ultra 9 285K (ARL), and an 8-core Core Ultra 7 258V (LNL CPU).Source: A. Experimental platforms · p. 6
  • EMR runs the AVX-512 kernel, while ARL and LNL run the AVX2 kernel.Source: A. Experimental platforms · p. 6
  • The GPU platforms are the integrated Intel Arc 140V and the discrete Intel Arc Pro B70, with measured read bandwidths of about 108 GB/s and 500 GB/s, respectively.Source: A. Experimental platforms · p. 6

B. Results on the CPU platforms

The CPU roofline model frames BITCOS performance as a competition between instruction throughput and memory bandwidth. Its implications vary by platform: bandwidth-bound cores can benefit from the smaller payload, while instruction-bound cores may not.

1) A roofline model for the BITCOS CPU kernels:

  • Emerald Rapids and Arrow Lake performance cores are bandwidth-bound, Arrow Lake efficiency cores sit at the knee, and Lunar Lake is instruction-bound.At zero density z = 0.40, Figure 10 places the measured core groups in these regimes.Source: Figure 10 · p. 7, 1) A roofline model for the BITCOS CPU… · p. 7
  • BITCOS throughput is limited by the lower of its instruction ceiling and memory-bandwidth limit, with the density-dependent byte count shifting the roofline knee.The model measures iteration cost with L1-resident operands and uses each platform’s per-core streaming read bandwidth.Source: 1) A roofline model for the BITCOS CPU… · pp. 6–7
  • On bandwidth-rich client platforms with few cores, cheaper 2-bit decoding can beat BITCOS despite its smaller payload.Lunar Lake leaves roughly three quarters of the bandwidth available per core unattainable for this kernel.Source: 1) A roofline model for the BITCOS CPU… · p. 7

2) GEMV microbenchmarks on Emerald Rapids:

BITCOS GEMV outperforms the 2-bit reference on Emerald Rapids and Arrow Lake across the deployed zero-density range. Both platforms convert most of the payload reduction into execution-time savings.

2) GEMV microbenchmarks on Emerald Rapids:

  • 1.14–1.28×: BITCOS GEMV speedup over the 2-bit SOTA GEMV on Emerald Rapids at deployed zero densities z ∈[0.297, 0.515].The kernel remains bandwidth-bound for z < 0.6 and sustains roughly 225 GB/s through z = 0.6.Source: 2) GEMV microbenchmarks on Emerald… · p. 7

3) GEMV microbenchmarks on Arrow Lake:

  • 1.13–1.27×: BITCOS GEMV speedup over the 2-bit SOTA GEMV on Arrow Lake at deployed zero densities z ∈[0.297, 0.515].The kernel sustains 93.4–93.6 GB/s of effective bandwidth across that density band.Source: 3) GEMV microbenchmarks on Arrow Lake · p. 7

4) GEMV microbenchmarks on Lunar Lake:

On CPUs, BITCOS’s end-to-end decode gains depend on the platform: memory-bound systems benefit, while instruction-bound Lunar Lake does not. The Lunar Lake GEMV results are consistent with that distinction.

4) GEMV microbenchmarks on Lunar Lake:

5) End-to-end decode on the CPU platforms:

  • 1.10–1.18× on Emerald Rapids and 1.02–1.15× on Arrow Lake: BITCOS end-to-end decode speedups over the SOTA 2-bit kernel, while Lunar Lake loses on every model.BITCOS is also 1.13–1.48× faster than TQ1_0 on Emerald Rapids and 1.46–1.74× faster on Arrow Lake; Figure 12 compares decode throughput across three CPU platforms and seven models.Source: 5) End-to-end decode on the CPU… · pp. 8–9, Figure 12 · p. 8

C. Results on the Xe2 GPU platforms

On the Arc 140V Xe2 integrated GPU, BITCOS GEMV beats the int2 reference across sampled zero densities. At deployed-model densities, the gains are smaller than the payload reduction because decode work does not fall with the payload.

1) GEMV microbenchmarks on the Arc 140V:

  • 1.04–1.14×: BITCOS GEMV speedup over the int2 state-of-the-art GEMV on Arc 140V at deployed zero densities z ∈[0.297, 0.515].The kernel delivers 71.7–75.6 GB/s of effective bandwidth across that band; every BITCOS point was tuned independently.Source: 1) GEMV microbenchmarks on the Arc 140V · p. 9
  • BITCOS wins at every sampled Arc 140V density, but its realized speedup is smaller than the payload reduction because decode work per weight stays constant as payload shrinks.Across the full sweep, effective bandwidth falls from 87.2 to 65.5 GB/s as zero density rises.Source: 1) GEMV microbenchmarks on the Arc 140V · p. 9

2) GEMV microbenchmarks on the Arc Pro B70:

On Xe2 GPUs, BITCOS outperforms the 2-bit reference in GEMV and end-to-end decode across the tested models. The section also situates these kernels among ternary training and quantization approaches.

2) GEMV microbenchmarks on the Arc Pro B70:

  • Across deployed zero densities, BITCOS GEMV is 1.01–1.12× faster than the int2 reference on the Arc Pro B70.Effective bandwidth is 398.7–421.9 GB/s across that density range; at z = 0.40, the 1.06× speedup falls short of the 1.23× byte ratio.Source: 2) GEMV microbenchmarks on the Arc Pro… · pp. 9–10

3) End-to-end decode on the GPU platforms:

  • BITCOS improves end-to-end decode 1.09–1.22× on the integrated Arc 140V and 1.02–1.27× on the discrete Arc Pro B70.Results use vLLM and single requests of 256 output tokens; BITCOS also outperforms llama.cpp Q2_0 by 1.11–1.18× and 1.61–2.30×, respectively.Source: 3) End-to-end decode on the GPU… · p. 10

A. Ternary Quantization of LLMs

  • QAT applies ternary-weight constraints during training, while PTQ quantizes pretrained models and calibrates on a few sequences.The section describes QAT and PTQ as complementary means of obtaining ternary models for inference.Source: A. Ternary Quantization of LLMs · p. 10

B. Kernels and runtimes for ternary LLM inference

The paper compares BITCOS with established ternary inference kernels and sparsity approaches, then concludes that its layout improves storage and inference performance while using existing checkpoints.

B. Kernels and runtimes for ternary LLM inference

  • LIBXSMM/XeTLA kernels are the paper's strongest published 2-bit baseline for CPU and Xe2 GPU platforms, and are equal to or faster than the best llama.cpp configurations.The authors attribute BITCOS's advantage over these roofline-optimal kernels to exploiting the inherent zero density of ternary LLM weights.Source: B. Kernels and runtimes for ternary… · p. 10

C. Use of sparsity in ternary LLM inference

  • Ternary CSC sparsity helps compute-bound prefill but performs worse than libTorch for batch-one decode, which is bandwidth-bound.Its layer-level kernels need 75–88% sparsity to beat dense cuBLAS, above the 29.7–51.5% found in most ternary checkpoints.Source: C. Use of sparsity in ternary LLM… · pp. 10–11
  • Unlike semi-structured sparsity methods that require training, BITCOS uses existing unstructured zeros and decodes the original ternary values bit-exactly.Sparse-BitNet trains with a dynamic N:M mask, while Sherry fixes zero density at 25% and enforces a 3:4 pattern.Source: C. Use of sparsity in ternary LLM… · p. 11

V. CONCLUSION

  • BITCOS improves end-to-end decode 1.10–1.18× on server CPUs, 1.02–1.15× on client CPUs, 1.09–1.22× on integrated GPUs, and 1.02–1.27× on discrete GPUs.These results are measured over seven ternary LLMs; BITCOS stores 26 of 29 checkpoints more compactly than five-trit packing.Source: §V · p. 11
Loading 2609.16338v1…