PaperlayerView exact revision on arXiv

Source-linked AI summary

BiLoCo: Binary Low-Rank Corrections for Post-Training Quantization

David Jin, Beshr IslamBouli, Tarushii Goel, Han Guo, Yoon Kim

arXiv:2610.32243v1cs.LG

TL;DR

Direct PTQ to FP4 can leave an accuracy gap, while high-precision low-rank corrections require substantial storage and may compete for tensor cores. BiLoCo replaces those factors with binary sign vectors so correction can run on CUDA cores alongside the FP4 GEMM. Across LLMs from 4B to 72B parameters, it matches or improves memory-matched SVDQuant, with batch-one fused-kernel latency overhead of 6–11% for ranks 16–512.

  • Problem

    Direct PTQ to FP4 leaves an accuracy gap, while BF16 low-rank corrections require more storage per rank and use tensor cores also needed by the main GEMM.

    Source: §1 · p. 1

  • Method

    BiLoCo stores low-rank correction factors as binary sign vectors with scalar coefficients, enabling signed add/subtract correction on CUDA cores.

    Source: §1 · p. 2

  • Results

    Across 4B–72B models, BiLoCo matches or improves SVDQuant at matched storage, and batch-one latency overhead is 6–11% for ranks 16–512.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    At the same vector-storage budget as BF16 rank 32, BiLoCo rank 512 captures 47–53% of weight Frobenius norm versus 26–29% for BF16 rank 32.

    Source: §1 · p. 2

    Latency claims are scoped to B300 and the evaluated projection shapes; BiLoCo targets linear weight projections, not attention operations or KV-cache traffic.

    Source: §6 · p. 10

Abstract

from arXiv · show

Post-training quantization (PTQ) to FP4 has emerged as a key technique for reducing large language model inference cost, especially on Blackwell GPUs, which offer native FP4 tensor core support. Direct PTQ to FP4, however, still leaves a noticeable accuracy gap. Recent work on SVDQuant mitigates the errors introduced through FP4 quantization for diffusion models by adding high-precision low-rank corrections obtained via SVD. However, this high-precision low-rank correction requires many bits per rank, and it runs on the same tensor cores as the main FP4 GEMM. We introduce BiLoCo, a low-rank correction method whose left and right factors are stored as binary ${\{\pm 1\}}$ sign vectors rather than as BF16 numbers. The signed format uses $16{\times}$ less storage per rank-one component, and since the correction now consists of additions/subtractions, this can be run on CUDA cores concurrently with the FP4 tensor-core GEMM. On 4B--72B LLMs, BiLoCo matches or improves memory-matched SVDQuant for PTQ. We implement BiLoCo on B300 as a fused single kernel, where the FP4 GEMM runs on tensor cores, the bit-packed correction on the CUDA cores of the same launch, and the two are combined in the GEMM epilogue. With batch size $1$, the fused kernel is at most $11\%$ slower than the FP4-only GEMM for every rank up to $512$. At rank $128$, it is at most $20\%$ slower up to batch size $8$.

1 Introduction

FP4 PTQ reduces LLM inference weight traffic but leaves an accuracy gap, while high-precision low-rank corrections use substantial storage and can compete for tensor cores. BiLoCo uses binary sign factors for a correction that runs on CUDA cores, and its fused Blackwell kernel preserves the reported memory-matched quality with modest latency overhead.

1 Introduction

  • LLM decode is dominated by weight movement, so tensor cores can stall on HBM while CUDA cores sit idle during the FP4 GEMM.A CUDA-core correction can therefore run concurrently instead of competing with the FP4 GEMM for tensor cores.Source: §1 · p. 1
  • Binary ±1 factors use 16× fewer bits per rank-one component than BF16 vector pairs and turn correction into CUDA-core additions and subtractions.At the same vector-storage budget, a BF16 rank-k branch can be replaced with a binary rank-16k branch.Source: §1 · p. 2
  • BiLoCo fits sign-vector components greedily, alternates updates to each factor, then jointly refits scales before quantizing the residual to NVFP4.The method starts each component from random signs and subtracts it from the residual after computing its best scale.Source: §1 · p. 2
  • 47–53% of LLM weight Frobenius norm is captured by BiLoCo rank 512 versus 26–29% by BF16 rank 32 at the same vector-storage budget.BiLoCo also matches or improves BF16 rank-32 correction perplexity across five models on WikiText-2/C4.Source: §1 · p. 2
  • At matched storage, BiLoCo matches or improves SVDQuant perplexity across five models, while batch-one latency rises 6–11% for ranks 16–512.For overhead within 20%, the largest batch size is 8 at rank 128, 4 at rank 256, and 2 at rank 512.Source: §1 · p. 2

2 Background

NVFP4 enables native low-precision tensor-core GEMMs, but its coarse codebook and shared block scales leave quantization error. Low-rank correction methods address that error, while GPU tensor and CUDA cores can issue work independently.

2.1 FP4 Quantization

  • NVFP4 quantizes blocks of 16 input elements to E2M1 using a shared E4M3 scale, with an additional tensor-wide FP32 scale.Its on-disk cost is approximately 4.5 bits per weight.Source: §2.1 · p. 3
  • Direct NVFP4 PTQ can leave an accuracy gap because coarse E2M1 values and one scale per block amplify errors from large entries and hidden-state outliers.The gap is especially relevant when both weights and activations are quantized.Source: §2.1 · p. 3

2.2 Low-Rank Corrections for Quantization Error

  • Low-rank corrections pair a quantized residual with higher-precision factors; SVDQuant uses BF16 factors, whose cost leads to higher-precision GEMMs.LQ-LoRA and LoftQ apply related decompositions for weight-only quantization; SVDQuant applies the approach to 4-bit weight-and-activation quantization for diffusion models.Source: §2.2 · p. 3

2.3 Tensor Cores and CUDA Cores

  • Tensor cores execute matrix multiply-accumulate instructions while CUDA cores handle scalar and vector instructions, and the two unit types issue independently.At LLM decode batch sizes, weight-reading limits the GEMM, leaving tensor cores underused and epilogue warps waiting during much of the main loop.Source: §2.3 · p. 3

3 Method

BiLoCo uses binary rank-one corrections to make more correction directions fit within a fixed storage budget, then combines them with an NVFP4 residual. Its method pairs greedy sign-vector fitting and a global coefficient refit with a fused kernel that runs the correction on CUDA cores alongside the FP4 GEMM.

3.1 Motivating Study: LLM Weights Favor Rank Multiplicity

  • At equal vector storage, signed rank 512 retains 47–53% of the Frobenius norm, versus 26–29% for BF16 SVD rank 32; binary factorization has higher capture at every tested budget.The rank-512 versus rank-32 comparison uses the same vector-storage budget; Figure 2 reports medians and interquartile ranges across 63 Qwen3-4B projection matrices.Source: Table 1 · p. 4, §3.1 · p. 4, Figure 2 · p. 4

3.2 BiLoCo: Binary Low-Rank Corrections for Quantized Weights

  • BiLoCo stores signed rank-one terms and scalar coefficients as the correction to an NVFP4 residual, rather than using a BF16 factorization.The sign vectors allow each inner product to use additions and subtractions instead of multiplications, and the factors take 16× less memory than BF16 factors.Source: §3.2 · p. 4
  • For each component, BiLoCo alternates sign updates from a random initialization, subtracts the scaled component from the residual, then jointly refits all coefficients by least squares before quantizing the residual.The joint refit accounts for overlap between components; sign alternation and the refit follow Riasanovsky and Kazdadi [16].Source: §3.2 · p. 5, §3.3 · p. 5

3.3 Theoretical Analysis

  • The analysis establishes monotone single-term sign updates and optimal coefficients for selected signs, while conditional residual decay requires a uniformly good signed-cut oracle.The authors explicitly do not claim that their alternating heuristic achieves a fixed approximation ratio to the optimal cut norm.Source: §3.3 · pp. 5–6

3.4 Kernel Design

  • The fused kernel runs FP4 GEMM on tensor cores and computes the bit-packed correction on CUDA cores, adding it in the GEMM epilogue without separate launches or memory passes.At rank 128, the correction reads about 0.16 MiB of signs plus the BF16 token, versus about 12.5 MiB of FP4 weights and 1.6 MiB of scales.Source: §3.4 · pp. 6–7
  • Correction overlap depends on batch size and rank: at batch 16 with rank at least 256, or batch 32 with rank at least 128, the correction runs before the GEMM rather than concurrently.For wide MLP projections where N/64 exceeds the SM count, the passage identifies a tensor-core correction as the better choice.Source: §3.4 · p. 7

4 Experiments

Across five LLMs, BiLoCo generally matches or improves memory-matched SVDQuant on perplexity, KL divergence, and downstream accuracy. Its fused B300 kernel adds modest latency at small decode batches, with overhead depending on rank and projection shape.

4 Experiments

  • Evaluation covers five LLMs, with calibration from 128 C4 training sequences and kernel timing on B300 at K=N=5120 across batch sizes M=1–32.Kernel ratios use the fastest padded FP4-only GEMM measured in the same run.Source: §4.1 · p. 7, §4.2 · p. 8

4.1 Performance Evaluation

  • At b=512, BiLoCo r=512 matches or improves SVDQuant k=32 on perplexity across five models, with ties on some model-dataset pairs.BiLoCo is slightly behind by at most 0.05 PPL at r=128, which uses a quarter of the storage.Source: §4.1 · p. 7, Table 2 · p. 8
  • At b=512, BiLoCo r=512 has lower 99th-percentile KL on all five models and lower mean KL on all but Qwen2.5-72B.On four models, the relative gap is larger at the 99th percentile; for Qwen3-14B, it is 0.130 vs. 0.167, 22% lower.Source: §4.1 · pp. 7–8, Table 3 · p. 8
  • At b=512, BiLoCo improves over SVDQuant on the four Qwen models by 0.1–0.7 points and is 0.3 lower on Llama-3-8B.At r=128, it is 0.4 points above SVDQuant k=32 on Qwen3-14B and at most 0.3 points below it on every model.Source: §4.1 · p. 8, Table 4 · p. 9

4.2 Decode Kernel Efficiency

  • On B300 (K=N=5120), at M=1 the correction adds 6–11% over the same-run padded FP4-only kernel for ranks 16–512; its within-20% batch range declines with rank, from M=32 at r=16 to M=2 at r=512.Source: §4.2 · p. 8, Table 5 · p. 9
  • At M=1, overhead is smallest at the down projection, where the FP4 GEMM is longest, and largest at the narrow k/v projection.The fused kernel does not support gate and up projections because their N/64=272 output tiles exceed the 148 SMs.Source: §4.2 · p. 9

5 Related Work

BiLoCo builds on low-rank correction and quantization work while changing the correction representation and execution path. The related work distinguishes this additive PTQ branch from training-time or fully binary models.

Post-training quantization for LLMs

  • Rotation-based W4A4 methods [2] [14] are orthogonal to correction format, and BiLoCo's binary branch can be applied on a rotated backbone.Appendix A.3 evaluates it with QuaRot and SpinQuant on Llama-3-8B.Source: §5 · p. 9

Low-rank corrections for FP4 quantization

  • SVDQuant introduced FP4 with a BF16 rank-32 correction for diffusion models; BiLoCo studies a binary correction at the same byte budget for LLMs.At that budget, BiLoCo uses 512 binary rank-one components in place of BF16 rank 32.Source: §5 · p. 9

1-bit and ternary networks

  • BitNet [20] and 1.58-bit LLMs [26] train binary or ternary weights from scratch, whereas BiLoCo adds a binary PTQ branch to an FP4 model.LQ-LoRA targets training-time storage rather than inference latency.Source: §5 · p. 9

Signed cut decompositions on LLM weights

  • Prior signed cut decompositions [16] share BiLoCo's per-component sign search and global coefficient refit, but BiLoCo uses the components as an FP4 additive branch on CUDA cores.This frames the method around matched storage and execution within the FP4 GEMM kernel.Source: §5 · p. 10

Spectral structure of LLM weights

  • At the SVDQuant rank, LLM weight spectra are flat enough that a fixed correction budget favors many binary components over fewer high-precision ones.The paper presents this as complementary to prior spectral analyses and SVD-based compression work [19] [21] [22] [24].Source: §5 · p. 10

Heterogeneous execution on GPUs

  • BiLoCo applies heterogeneous GPU execution within a projection layer, assigning the residual GEMM to FP4 tensor cores and the binary correction to CUDA cores.This follows a broader pattern of overlapping matrix multiplication, memory movement, and other work in GPU kernels [13] [18].Source: §5 · p. 10

6 Limitations

The fused-kernel latency evidence is scoped to B300 projection shapes and measured batch sizes. Rotation experiments cover only one model and seed.

Kernel scope

  • Latency claims are limited to B300 and the projection shapes in Tables 5 and 6.The KV cache remains BF16; BiLoCo targets linear weight projections and their activations, not attention operations or KV-cache traffic.Source: §6 · p. 10
  • The fused kernel stays within 20% of FP4-only latency through M=8 at r≤128, M=4 at r=256, and M=2 at r=512.Larger batches and wide MLP projections whose tile count exceeds the SM count are better served by a tensor-core correction.Source: §6 · p. 10

Composition with rotations

  • Rotation comparisons are limited to one model and seed, where both tested techniques underperform presmoothing under NVFP4 block quantization.Source: §6 · p. 10

7 Conclusion

BiLoCo combines one-bit low-rank sign factors with an FP4 residual, matching or surpassing memory-matched SVDQuant across 4B–72B models. Its binary correction runs on CUDA cores alongside the tensor-core GEMM in a fused Blackwell kernel, with small-batch latency close to FP4-only.

7 Conclusion

  • Across models from 4B to 72B parameters, BiLoCo matches or surpasses SVDQuant at the same storage budget.The method splits each LLM weight matrix into one-bit sign factors and an FP4 residual.Source: §7 · p. 10
  • Binary factors reduce the correction arithmetic to additions and subtractions, enabling CUDA-core execution concurrently with the tensor-core GEMM.The fused Blackwell kernel computes the low-rank product from packed signs and adds it in the GEMM epilogue.Source: §7 · p. 10
  • At small decode batch sizes, the fused kernel stays close to baseline FP4 latency while recovering quantization accuracy.Source: §7 · p. 10

A.1 Structural and cut-premise diagnostics

The appendix examines how correction capture varies across projection types and tests an empirical proxy for a theorem's cut-decay premise. The measured proxy stays bounded away from zero across the diagnostic sweep.

Structural diagnostics

  • The projection-type comparison reports median Frobenius capture across nine evenly spaced decoder layers per model, using truncated SVD and greedy signed-peel reconstructions.The reported binary advantage holds for every projection type on all three models.Source: §A.1 · p. 13
  • BiLoCo rank 256 captures more than SVDQuant rank 32 on all 21 projection–model pairs while using half the storage.At a quarter of the storage, BiLoCo captures about the same fraction on Qwen3-14B and Llama-3-8B and substantially more on Qwen3-4B.Source: §A.1 · p. 13

Cut-premise diagnostics

  • Across 8,064 rank-one updates, ρF is at least 0.019, with its median declining from 0.036 at iteration 0 to 0.026 at iteration 127.The diagnostic uses a Frobenius analog because the theorem's cut-norm premise is NP-hard to compute; the analog lower-bounds that condition.Source: §A.1 · p. 13

A.2 Solver ablation, construction cost, and perplexity across ranks

The appendix compares BiLoCo construction methods and costs, then examines how correction rank affects perplexity and compatibility with rotated backbones. The results show that learned signs matter, higher ranks continue to improve perplexity, and BiLoCo improves both tested rotated backbones over memory-matched SVDQuant.

A.2 Solver ablation, construction cost, and perplexity across ranks

  • Learned signs provide the gain: random signs with an optimal joint coefficient refit yield WikiText-2 perplexity 7.000, versus 6.998 for uncorrected NVFP4.Greedy peeling, global refit, and pm-altmin are within seed noise of one another at r=512.Source: §A.2 · p. 13, Table 8 · p. 14
  • 86 s builds 512 greedy binary components, half the 170 s rank-32 SVD construction time it replaces.The global coefficient refit adds about 5 s; residual quantization and bookkeeping take under a second, while full alternating minimization is an order of magnitude more expensive and not the default.Source: §A.2 · pp. 13–14, Table 9 · p. 14
  • Perplexity improves monotonically with rank and continues past the matched-storage point; r=1024 and r=2048 improve further at 2× and 4× SVDQuant k=32 storage.The authors set r=512 by the byte budget rather than accuracy, while noting that kernel overhead grows with rank.Source: §A.2 · p. 14, Table 10 · p. 15

A.3 Composition with Rotated Backbones

  • BiLoCo improves both rotated backbones and beats same-storage SVDQuant on QuaRot and SpinQuant, although rotation itself hurts under NVFP4.The authors attribute the NVFP4 result to rotation spreading localized outliers across 16-element scaling blocks; the experiment uses one model and one seed.Source: §A.3 · p. 14, Table 11 · p. 15

A.4 Full Downstream Task Table

The downstream-task table compares BiLoCo ranks across three models, while the KL table evaluates quantized-model token distributions against a BF16 reference. BiLoCo’s five-task mean is higher at r=512 on two models, and its b=512 KL is lower than SVDQuant’s on all three reported models.

A.4 Full Downstream Task Table

  • BiLoCo r=512 has a higher five-task mean on Qwen3-4B and Qwen3-14B; on Llama-3-8B it is 0.3 points lower, while r=128 has the highest mean.The tasks are ARC-easy, ARC-challenge, PIQA, OBQA, and WinoGrande.Source: §A.4 · p. 15, Table 12 · p. 15

A.5 Full KL Table

  • At b=512, mean KL is lower for BiLoCo than SVDQuant on Qwen3-4B (0.074 vs. 0.084), Qwen3-14B (0.048 vs. 0.052), and Llama-3-8B (0.076 vs. 0.082).BiLoCo also has lower KLp99 on all three models; at b=128, results are mixed.Source: §A.5 · p. 15, Table 13 · p. 16

A.6 Fused-Kernel Implementation

The fused kernel combines the FP4 GEMM and a binary correction in one launch, using CUDA cores for correction work and the GEMM epilogue to add it. Its scheduling changes with batch size and rank, and an equal-storage control separates format from fusion effects.

A.6 Fused-Kernel Implementation

  • At small M, padding the FP4 activation to the full 128-row tile avoids a 23–36% slowdown from repeated reads of the small A tile.The correction is computed only for real rows, and the FP4-only baseline uses the same padding.Source: §A.6 · p. 16
  • At K=N=5120, side CTAs compute the binary-factor intermediate on CUDA cores while GEMM tensor cores run; epilogue warps expand and add the correction before the BF16 store.For larger Mr, all CTAs compute the correction at kernel entry, so it is no longer hidden under the GEMM.Source: §A.6 · p. 16
  • At M=1 and K=N=5120, dense BF16 rank 32 and binary rank 32 each take 6.20 µs, while binary rank 512 takes 6.49 µs and improves accuracy over BF16 rank 32.The unfused dense correction takes 17.64 µs; the authors attribute binary’s advantage to the rank available at a fixed storage budget, not faster equal-rank arithmetic.Source: §A.6 · p. 16

B Proofs

The appendix provides formal results for sign alternation, conditional residual decay, and coefficient refitting. Together, these statements characterize the single-term updates, a conditional peeling guarantee, and when the refit has a unique solution.

B.1 Monotone Single-term Sign Alternation

  • Each sign update does not decrease the single-term objective, and deterministic tie-breaking over the finite state space yields convergence to a fixed point or plateau.For fixed u, the v update maximizes u⊤Rv over sign vectors; the symmetric statement holds for the u update.Source: §B.1 · p. 16
  • For a fixed pair of sign vectors, the best scale minimizes the residual Frobenius norm and equals u⊤Rv/(mn).The resulting identity gives the Frobenius-norm decrease from subtracting that rank-one component.Source: §B.1 · pp. 16–17

B.2 Conditional Signed-cut Decay

  • The conditional signed-cut result gives a minimum-over-iterations residual bound; it does not claim monotonicity of the final cut norm.The bound is conditional on the selected signs satisfying the theorem’s stated condition.Source: §B.2 · p. 17

B.3 Optimal Coefficient Refit

  • The coefficient-refit matrix is a Gram matrix and therefore positive semidefinite; linear independence of the terms makes it positive definite and the minimizer unique.The implemented solve adds a numerical ridge, using (A + ϵI)−1b.Source: §B.3 · p. 17
Loading 2610.32243v1…