PaperlayerView exact revision on arXiv

Source-linked AI summary

FluxBin: Flexible LUT-based Ultra-low-bit LLM Inference by Algorithm-Kernel Synergy

Qingyao Yang, Runming Yang, He Xiao, Wendong Xu, Junyu Chen, Haobo Liu, Chenchen Ding, Ruihan Hu, Yik-Chung Wu, Ngai Wong

arXiv:2608.15602v1cs.LGcs.AI

TL;DR

Existing binary quantization methods struggle to convert improved fidelity into practical end-to-end speedups. FluxBin co-designs post-training quantization and a LUT-based CUDA kernel, achieving up to 5.92× speedup and 10.19× energy savings across diverse architectures while maintaining competitive accuracy.

  • Problem

    Existing multi-binary quantization methods improve fidelity but still struggle to translate binary representations into practical end-to-end inference speedups.

  • Method

    FluxBin co-designs pure post-training quantization using decoupled row-column binary bases and Hessian-guided hybrid bases with a LUT-based CUDA kernel.

  • Results

    Across diverse architectures, FluxBin achieves up to 5.92× speedup and 10.19× energy savings while maintaining competitive accuracy.

  • Takeaways & Limitations

    FluxBin offers a training-free approach for efficient large-scale LLM deployment.

  • Takeaways & Limitations

    FluxBin’s LUT-kernel efficiency depends on shared memory, and limited-SRAM hardware may require reducing sub-vector size or batch size.

Abstract

from arXiv · show

While binary quantization theoretically promises extreme compression and acceleration for Large Language Models (LLMs), existing research often overlooks the necessity of specialized hardware kernels, thus failing to unleash the full acceleration potential due to persistent reliance on expensive floating-point arithmetic or runtime dequantization overheads. To bridge this gap, we propose FluxBin (\textbf{F}lexible \textbf{L}UT-based \textbf{U}ltra-low-bit e\textbf{X}ecution with \textbf{Bin}ary bases), an algorithm-kernel co-design that synergizes post-training quantization with a highly optimized CUDA kernel. Algorithmically, we introduce Decoupled Row-Column Binary Decomposition to enhance representational capacity while maintaining hardware efficiency, complemented by a Hessian-guided saliency-aware hybrid bases that preserve critical information. At the kernel level, we implement a Lookup Table Building Approach with Scale Fusion to reduce floating-point arithmetic, featuring a Virtual Columnar Mapping that transforms irregular, sparse, and salient matrices into dense execution. Extensive evaluations demonstrate FluxBin achieves up to $5.92\times$ speedup and $10.19\times$ energy savings across diverse model architectures, delivering comparable accuracy to heavily fine-tuned methods. This effectively enables the deployment of 70B-scale models on one single A100 GPU with a $4\times$ memory reduction. Code is available at https://github.com/nicyyyy/FluxBin.

1 Introduction

FluxBin addresses the gap between binary quantization’s theoretical efficiency and practical LLM inference by co-designing a flexible PTQ method with an optimized LUT-based CUDA kernel. Its approach combines decoupled binary decomposition, Hessian-guided hybrid bases, and dequantization-free execution to improve accuracy and achieve substantial speed and energy gains.

  • Motivation: Post-training quantization reduces bit-width without retraining, while binary quantization offers maximal theoretical compression efficiency.Binary representations are motivated by severe LLM storage and bandwidth constraints.
  • Motivation: Multi-binary methods improve fidelity by approximating weights with combinations of binary matrices, but struggle to deliver practical end-to-end inference speedups.The introduction identifies this limitation in QBB, PTQTP, and DB-LLM.
  • Limitations: Existing approaches face multiplication-free hardware underuse, saliency granularity–efficiency trade-offs, and low-bit kernel inefficiencies from irregularity or runtime dequantization.Unstructured sparsity creates irregular memory access and metadata overhead, while W4A16 methods convert weights to FP16 before multiplication.
  • FluxBin: FluxBin introduces decoupled row-column binary decomposition and Hessian-guided hybrid bases within a pure PTQ framework to balance hardware efficiency with competitive accuracy.The framework is designed to capture fine-grained distributions while preserving critical information.
  • FluxBin: FluxBin’s LUT-based CUDA kernel uses LUT-BSF and VCM strategies to support high-fidelity hybrid precision and turn binary PTQ into practical inference gains.The kernel is presented as the hardware counterpart to the algorithmic design.
  • Results: 5.92× speedup and 10.19× energy saving are achieved by bypassing dequantization overhead and converting multiplication-free logic into practical acceleration.These are the maximum reported gains in the introduction.

2 Related Work

Prior work spans standard binary quantization, multi-basis binary approximation, and LUT-based multiplication for reducing arithmetic redundancy. However, existing LUT kernels do not provide PTQ adaptation, mixed-precision configurations, or end-to-end inference pipelines.

  • Binary Quantization: Standard binary quantization approximates W with a scaled binary matrix, typically using B = sign(W) and α = 1/(MN)||W||_1.ARB-LLM adds column-wise grouping and alternating refined binarization, while HB-LLM applies distinct strategies to wavelet-separated frequency components.
  • Binary Base Approximation: High-order binary quantization uses K binary bases to overcome the limited representation capacity of a single 1-bit basis.LQ-Nets jointly train bases and encodings, QBB greedily approximates residuals, and DB-LLM uses primary and secondary bases.
  • LUT-based Multiplication: LUT-based multiplication reduces redundant binary MVM computation by precomputing all 2^µ linear combinations for each input sub-vector and fetching partial sums by binary patterns.Repeated weight sub-patterns otherwise cause identical input sub-vector dot products to be computed multiple times.
  • LUT-based Multiplication: LUT-GEMM provides an efficient kernel for uniform quantization but lacks PTQ algorithm adaptation, mixed-precision configurations, and end-to-end inference pipelines.Recent LUT Tensor Core work further introduced an instruction set with elaborated precomputation and tiling.

3 Methodology

FluxBin quantizes weights with decoupled row-column binary bases, optimized through alternating minimization and saliency-aware hybrid bases. Its CUDA kernel fuses scaling into lookup tables, performs multiplication-free GEMV, and uses virtual columnar mapping to execute sparse salient refinements as dense operations.

  • Quantization: Decoupled row-column binary decomposition uses row and column scaling vectors with binary bases to capture anisotropic weight structure.The final quantized matrix sums approximations across decomposition orders.
  • Parameter Optimization: FluxBin optimizes scaling vectors and binary bases by minimizing reconstruction error through alternating minimization and weighted least squares.For multiple bases, it jointly evaluates all 2^K_b sign combinations to select locally optimal binary assignments.
  • Structural Saliency Search: Hessian-guided structural saliency ranks columns using inverse-Hessian sensitivity and selects salient columns within fixed-size groups for load-balanced refinement.The selected columns define a binary salient mask used by the hybrid quantization strategy.
  • Salient-Aware Hybrid Bases: The hybrid quantizer combines a global base approximation with a salient refinement applied to selected columns.The salient branch uses the same quantization function with its own binary bases and scaling parameters, while refining the global residual.
  • LUT-Based GEMV Kernel: The kernel builds scale-fused lookup tables, reads compressed weights with bitwise operations, and applies row scaling during register-level reduction.Fusing column scaling into LUT construction removes related floating-point operations from the main GEMV loop.
  • Virtual Columnar Mapping: Virtual Columnar Mapping remaps logically sparse salient columns into a contiguous dense matrix, enabling physically dense GEMV and kernel reuse.A translation layer gathers the corresponding activations and scaling factors at runtime.

4 Experiment

Experiments show that FluxBin delivers strong throughput, memory, energy, and architectural-scaling benefits while retaining competitive accuracy without training. Ablations further validate the importance of saliency-aware components, VCM, and key hyperparameter choices.

  • Memory Efficiency: 4× memory reduction enables LLaMA-2-70B execution on a single A100, whereas the FP16 baseline encounters out-of-memory errors.The reported compression capability supports deployment of massive models within one GPU’s memory.
  • Energy Efficiency: 10.19× energy reduction versus FP16 accompanies lower inference power, including a drop from ∼225W to ∼115W on 13B.The reduction is attributed to minimized data movement and less floating-point arithmetic.
  • Architectural Robustness and Generalization: FluxBin generalizes across Qwen3 and LLaMA-3.1, maintaining consistent acceleration across attention mechanisms and model sizes.On sensitive LLaMA-3.1 models, it achieves accuracy comparable to 3-bit baselines and exceeds AQLM by +17.5% accuracy.
  • Competitive Accuracy without Training: 59.15% average zero-shot accuracy: FluxBin-2b-s16-g128 matches QuIP# at 59.41% and exceeds AQLM at 58.64% without training.This result demonstrates competitive accuracy relative to fine-tuning- or distillation-based approaches.
  • Throughput and Scalability: 5.92× speedup: FluxBin reaches 254.39 tokens/s on LLaMA-2-7B versus FP16, and achieves 3.25× higher throughput than QuIP# on LLaMA-2-70B.Hybrid configurations retain speed advantages across batch sizes because LUT computation reduction is batch-size independent.
  • Ablation and Sensitivity Analysis: 18% speed drop and 33% memory surge: disabling VCM validates its role in enabling coalesced memory access for refinement operations.Ablations also show that random saliency selection causes PPL 26.02, while using only the Global Base raises PPL to 13.71.

5 Conclusion

FluxBin bridges theoretical compression and realized acceleration by combining binary decomposition, Hessian-guided hybrid bases, and an optimized LUT-based kernel. Across diverse architectures, it delivers up to 5.92× speedup and 10.19× energy savings while maintaining competitive accuracy through a training-free approach.

  • 5 Conclusion: FluxBin combines Row-Column Binary Decomposition, Hessian-guided Hybrid Bases, and a hardware-optimized LUT-based Kernel to improve capacity and reduce runtime overhead.The kernel includes LUT-BSF and VCM.
  • 5 Conclusion: 5.92× speedup and 10.19× energy savings are achieved across diverse architectures while maintaining competitive accuracy.FluxBin is presented as a training-free solution for efficient inference.

Limitations · A Derivation of Parameter Optimization · A.1 Derivation of Optimal Scaling Factors

FluxBin’s limitations concern shared-memory scalability and the accuracy ceiling of strict training-free PTQ. The appendix then provides mathematical proofs for parameter updates, deriving optimal row and column scaling factors through decoupled least-squares optimization.

  • Limitations: FluxBin identifies two primary limitations despite its advances in ultra-low-bit LLM inference.
  • Limitations: The LUT-based kernel’s shared-memory requirement scales exponentially with sub-vector size 2^µ and linearly with batch size per block.The A100 provides 164 KB/SM and supports the optimal configuration µ = 8; hardware with less shared memory may require reducing µ.
  • Limitations: Strict training-free PTQ leaves a marginal accuracy gap relative to computationally expensive iterative fine-tuning on specific tasks.The paper suggests lightweight parameter-efficient fine-tuning on binary bases as future work.
  • A Derivation of Parameter Optimization: The appendix provides rigorous mathematical proofs for the optimal update rules introduced in the Methodology section.
  • A.1 Derivation of Optimal Scaling Factors: For a single decomposition order k, the appendix recalls the corresponding objective function before deriving optimal scaling factors.
  • A.1 Derivation of Optimal Scaling Factors: Keeping B and αc fixed, the proof derives a closed-form solution for the row scaling vector αr by expanding the Frobenius-norm objective.
  • A.1 Derivation of Optimal Scaling Factors: Optimization of αr decouples into M independent scalar least-squares problems, one for each row scalar αr,i.
  • A.1 Derivation of Optimal Scaling Factors: For each αr,i, differentiating the objective, setting the derivative to zero, and solving yields the row-scale expression.

A.2 Derivation of Joint Binary Basis Optimization … C.1 Compute Complexity Analysis of LUT-base Kernel

FluxBin proves that its element-wise binary assignment is globally optimal for fixed scaling factors, then integrates this optimization with GPTQ-based saliency handling and a LUT kernel whose lookup cost dominates runtime and yields a theoretical complexity reduction over FP16 multiplication.

  • A.2 Derivation of Joint Binary Basis Optimization: Because binary basis elements are independent across spatial coordinates, minimizing global Frobenius error decomposes into independent per-element problems.This reduction establishes the basis for solving each coordinate separately.
  • A.2 Derivation of Joint Binary Basis Optimization: For each coordinate, the optimization selects a binary-choice vector b ∈ {−1, +1}^Kb to minimize reconstruction error.The scalar problem is obtained by expressing the coordinate’s basis choices as a binary vector.
  • A.2 Derivation of Joint Binary Basis Optimization: Element-wise exhaustive search yields the globally optimal binary assignment when scaling factors are fixed.The proof uses equivalence between minimizing squared L2 error and absolute L1 error in the scalar setting.
  • B Algorithmic Descriptions: FluxBin integrates with GPTQ using block-wise reconstruction, inverse-Hessian saliency search, and iterative error compensation across columns.The Hessian is computed as H = 2XXT from calibration data, with its inverse updated iteratively.
  • B Algorithmic Descriptions: The Base variant applies Row-Column Binary Decomposition globally for pure 2b-g128 (Kb = 2) and 4b-g128 (Kb = 4) configurations.Residual error is propagated to remaining unquantized weights using the Hessian inverse.
  • B Algorithmic Descriptions: The Hybrid Base variant identifies critical columns with StructuralSearch and decomposes residuals on salient columns for 2b-s8-g128 (s = 8) and 2b-s16-g128 (s = 16).The secondary decomposition is restricted to the masked salient columns.
  • C.1 Compute Complexity Analysis of LUT-base Kernel: LUT-kernel cost comprises LUT construction and table lookup, with independent LUTs required for each basis because column scaling factors are basis-specific and fused into the input.The global branch uses N input dimensions, while salient refinement operates on Nsal columns.
  • C.1 Compute Complexity Analysis of LUT-base Kernel: M ≫2µ makes lookup cost dominate runtime; compared with FP16 O(MN), FluxBin provides an approximate reduction factor of µ Kb, while Nsal ≪N keeps hybrid overhead marginal.The total cost combines global and salient branches, represented by LUT-build and LUT-read terms involving KbN and KsNsal.

C.2 Storage Consumption Analysis

FluxBin’s storage analysis decomposes total storage into global-base and salient-refinement components, then derives effective average bit-width from their aggregation. In the typical configuration, salient refinement affects 6.25% of columns, while indexing is amortized over the matrix and ABW reaches approximately 2.63 bits.

  • Storage decomposition: The configuration groups columns into size g and selects s salient columns per group, defining the salient subset as N_sal = N/g · s.The quantization model uses K_b global bases and K_s salient-refinement bases, while scaling factors and indices are stored in FP16.
  • Storage decomposition: Total storage is modeled as the sum of global-base storage and salient-refinement storage, with effective average bit-width derived from both components.The global base stores dense binary weights, column-wise scales, and block-wise row scales; salient refinement stores sparse binary refinements, scales, and indexing metadata.
  • Typical configuration: 6.25% is the salient-branch sparsity ratio for K_b = 2, K_s = 2, g = 128, and s = 8.The indexing overhead is amortized over the large matrix, reducing its impact on effective storage.
  • Typical configuration: ≈2.63 bits is the effective ABW for the typical configuration, rising to ≈2.75 bits for s = 16 while boosting model accuracy.The reported ABW incorporates indexing overhead and is measured in bits per weight.

C.3 Maximum Memory Consumption · D Extension Experiment · D.1 Extension on Throughput

FluxBin substantially lowers inference memory while preserving throughput across binary and hybrid-precision configurations. Extended evaluations show robustness across GQA architectures and scalability to massive models, including successful 70B inference within 20.2GB.

  • C.3 Maximum Memory Consumption: On Qwen3-32B, memory drops from 62.7GB (FP16) to 12.1GB (2b-g128), achieving a 5.1×compression ratio.For LLaMA-2-70B, FP16 causes OOM, whereas FluxBin enables inference using 20.2GB in the 2-bit configuration.
  • C.3 Maximum Memory Consumption: For LLaMA-2-7B, increasing precision from 2-bit to 4-bit raises memory usage from 2.9GB (2b-g128) to 4.7GB (4b-g128).The increase reflects storage for additional binary bases.
  • C.3 Maximum Memory Consumption: Salient refinement adds moderate memory overhead, while the difference between 2b-s8 and 2b-s16 is negligible, below 1MB for LLaMA-2-7B.Virtual Columnar Mapping makes doubling salient columns minimally costly by efficiently storing auxiliary matrices and indices.
  • D Extension Experiment: The extension evaluates throughput on Qwen3 models at 8B, 14B, and 32B parameters and LLaMA-3.1 models at 8B and 70B.Qwen3 represents GQA architectures, while LLaMA-3.1 represents a latest-generation massive-scale family.
  • D.1 Extension on Throughput: Hybrid configurations 2b-s8-g128 and 2b-s16-g128 remain virtually indistinguishable in throughput from the pure 2b-g128 baseline across all evaluated cases.This indicates that salient refinement does not materially reduce throughput relative to the pure 2-bit configuration.
  • D.1 Extension on Throughput: FluxBin is architecture-agnostic, integrating with standard architectures and GQA designs such as Qwen3 while delivering consistent acceleration.The kernel operates without being hindered by specific architectural variations.
  • D.1 Extension on Throughput: On LLaMA-3.1-70B, FluxBin maintains the same high-efficiency profile as on smaller models, demonstrating scalability to massive models.The mixed-precision strategy’s overhead does not grow disproportionately with model size.

D.2 Extended Evaluation on Speed vs. Accuracy · D.3 Impact of Group Size

FluxBin generalizes across LLaMA-3.1 and Qwen3 architectures, retaining accuracy at roughly 2.75 bits while preserving binary-kernel throughput advantages. Group-size analysis shows hybrid salient-aware bases remain robust, with g=128 providing the preferred accuracy–bit-width balance.

  • D.2 Extended Evaluation on Speed vs. Accuracy: 36.18% and 50.27% average accuracy on LLaMA-3.1 70B show severe degradation for FlexRound and AQLM, respectively.FluxBin instead maintains exceptional fidelity with its 2b-s16-g128 configuration.
  • D.2 Extended Evaluation on Speed vs. Accuracy: +18.3% accuracy on Qwen3 8B Avg is achieved by FluxBin over the SLiM-LLM baseline.The evaluation targets Qwen3 architectures using Grouped-Query Attention.
  • D.2 Extended Evaluation on Speed vs. Accuracy: 69.39% vs. 70.45% on Qwen3 14B shows FluxBin’s roughly 2.75-bit accuracy competitively approaches the 3.125-bit GPTQ baseline.The comparison supports architecture-agnostic operation across modern LLM designs while retaining theoretical binary-kernel throughput advantages.
  • D.3 Impact of Group Size: 57.96% to 47.20% average accuracy occurs when pure 2b group size increases from g=64 to g=256.This sensitivity indicates coarse-grained global binary approximation struggles with local weight variance without refinement.
  • D.3 Impact of Group Size: 58.87% average accuracy is maintained by 2b-s16 at g=256, exceeding pure 2b at g=64 with 57.96%.Salient refinement compensates for information loss from coarse-grained grouping, and 2b-s8 shows the same robustness trend.
  • D.3 Impact of Group Size: 59.15% accuracy at g=128 provides the selected balance between performance and compact representation.The default setting uses an effective bit-width of ∼2.75 bits, whereas g=64 reaches the highest accuracy but inflates s16 to ∼3.5 bits.

D.4 Sensitivity to Calibration Sample Size · E Profiling Study · E.1 Energy Efficiency and Resource Utilization

Calibration size produces a rise-then-fall accuracy pattern, with Ncal = 256 offering the best accuracy–time trade-off. FluxBin profiling shows major energy savings and sustained GPU occupancy through reduced latency, power draw, and memory movement.

  • D.4 Sensitivity to Calibration Sample Size: Ncal from 64 to 1024 produces a distinct rise-then-fall trend in FluxBin model performance.
  • D.4 Sensitivity to Calibration Sample Size: 57.86% → 58.77% average accuracy and 8.92 → 8.63 Wiki PPL result when Ncal increases from 64 to 256.The larger calibration set provides a more stable Hessian estimate and better identifies sensitive weights.
  • D.4 Sensitivity to Calibration Sample Size: 58.07% average accuracy at Ncal = 1024 reflects calibration overfitting that harms generalization to the test distribution.
  • D.4 Sensitivity to Calibration Sample Size: ∼25 min at Ncal = 256 versus ∼75 min at Ncal = 1024 makes Ncal = 256 the selected default for the main experiments.PTQ time increases linearly with calibration sample size.
  • E.1 Energy Efficiency and Resource Utilization: Steady-state A100 profiling uses nvidia-smi at 100ms intervals during decode generation with 128-token input and output lengths and batch size 1.The first 10s of warmup are excluded, and total energy is computed by integrating power over time.
  • E.1 Energy Efficiency and Resource Utilization: 10.19× lower total energy for LLaMA-2-13B than the FP16 baseline, at 579.23J versus 5908.03J.
  • E.1 Energy Efficiency and Resource Utilization: ∼115W versus 225W average power draw contributes to FluxBin’s energy reduction alongside lower inference latency.Reduced data movement and lightweight bitwise/integer operations replace power-intensive FP16 arithmetic.
  • E.1 Energy Efficiency and Resource Utilization: >99.8% GPU busy time for FluxBin versus 67%–86% for FP16 indicates continuously occupied SMs without HBM-traffic stalls.The reported busy fraction is not arithmetic-unit saturation; low memory utilization supports the interpretation of efficient LUT-kernel execution.

E.2 Shared Memory Footprint Analysis · E.3 Tiling Strategy and TFLOPS Comparison · E.4 Kernel-Level Latency Comparison

FluxBin’s resource analysis supports μ = 8 as the default despite higher shared-memory use, while tiling choices improve throughput and its isolated 2-bit kernel outperforms AQLM and QuIP# on representative GEMV shapes.

  • E.2 Shared Memory Footprint Analysis: Shared-memory usage comprises LUT storage for batch-specific, scale-fused tables and an accumulation buffer holding partial sums for each row and binary base.The default tile configuration is Mtile = 128 and Ktile = 64.
  • E.2 Shared Memory Footprint Analysis: At BS 8, μ = 8 consumes 136 KB of shared memory, remaining below the NVIDIA A100’s 164 KB per-SM capacity despite LUT growth from 2^4 to 2^8.μ = 4 maintains a footprint below 24 KB.
  • E.2 Shared Memory Footprint Analysis: Although μ = 4 and μ = 8 deliver comparable end-to-end throughput, μ = 8 is selected because halving μ doubles sub-vectors and increases LUT reads 2×.The additional LUT-read frequency is reported to dominate the relevant complexity trade-off.
  • E.3 Tiling Strategy and TFLOPS Comparison: Increasing Mtile from 32 to 128 at K = 4096 raises 2-bit throughput from 0.30 TFLOPS to 0.81 TFLOPS by hiding instruction latency and amortizing LUT construction.Effective TFLOPS represents equivalent FP16 computational throughput, including salient-branch workload for hybrid configurations.
  • E.3 Tiling Strategy and TFLOPS Comparison: Ktile balances reduction efficiency against shared-memory occupancy, with empirical results indicating Ktile = 32 or 64 typically offers the optimal balance.Larger Ktile reduces global atomic reductions but consumes more LUT storage.
  • E.3 Tiling Strategy and TFLOPS Comparison: For hybrid bases, Mtile = 128 consistently performs best, while Ms_tile = 64 slightly reduces latency versus 16 through better dense-matrix memory coalescing.Virtual Columnar Mapping keeps scaling the salient branch from incurring substantial computational penalties.
  • E.4 Kernel-Level Latency Comparison: At 4096 × 4096, FluxBin’s isolated 2-bit GEMV kernel requires 41 µs, compared with 112 µs for QuIP# and 79 µs for AQLM.The comparison uses official implementations and isolates per-operator kernel latency across weight shapes dominating LLM linear layers.
Loading 2608.15602v1…