PaperlayerView exact revision on arXiv

Source-linked AI summary

FIGFLEX: A Mixed-Precision FP-INT GEMM Accelerator Co-Designed with Low-Bit LLM Quantization

Faraz Tahmasebi, Haocheng Xu, Andrew Trauger, Yomal Perera, Zhenyu Tang, Nael Chowdhury, Sitao Huang

arXiv:2610.30480v1cs.ARcs.LG

TL;DR

Low-bit mixed-precision LLM quantization reduces weight storage, but GPUs and fixed-precision bit-parallel accelerators do not make fewer weight bits reduce compute cost. FIGFLEX co-designs a precision-flexible FP-INT datapath and quantizer; on LLaMA-2-7B it reports lower WikiText-2 perplexity than the compared methods and higher throughput per area than AxCore and FIGLUT on 3-bit linear GEMMs.

  • Problem

    Mixed-precision weight quantization reaches low average bit-widths, but GPUs dequantize weights before computation and fixed-precision bit-parallel accelerators cannot exploit the varying group widths.

    Source: §I · pp. 1–2

  • Method

    FIGFLEX combines a radix-4 bit-parallel FP-INT datapath with a quantizer that assigns group precision according to the error GPTQ compensation leaves behind.

    Source: §I · p. 2, B. Radix-4 FP-INT Datapath · p. 4, C. Compensation-Aware Assignment · p. 6

  • Results

    On LLaMA-2-7B, FIGFLEX reports 5.85 and 8.77 WikiText-2 perplexity at 3- and 2-bit average precision, and 1.56× and 1.42× whole-chip throughput per area versus AxCore and FIGLUT on 3-bit linear GEMMs.

    Source: §VII · p. 12

  • Takeaways & Limitations

    FIGFLEX makes lower weight precision translate into higher bit-parallel compute throughput while retaining the reported low-bit model quality.

    Source: §VII · p. 12

Abstract

from arXiv · show

Weight-only quantization keeps LLM activations in floating point and stores weights as low-bit integers, turning the dominant workload into floating-point-integer (FP-INT) general matrix multiplication. Recent quantizers reach a 2-3-bit average with mixed precision, but hardware has not kept pace: GPUs dequantize weights to FP16 before the tensor-core multiply-accumulate, and bit-parallel FP-INT accelerators run at a single weight precision, so fewer bits save memory but not computation. In this paper, we present FIGFLEX, an algorithm-hardware co-designed mixed-precision accelerator whose bit-parallel datapath scales throughput as precision falls. FIGFLEX contributes (1) a shift-add, radix-4 reformulation of FP-INT multiplication that fuses the multiply-accumulate into a unified, precision-flexible partial-product reduction and roughly halves the partial products, together with a novel INT3 optimization that reclaims the idle datapath radix-4 leaves at that precision; and (2) a mixed-precision quantizer that assigns each channel group its bit-width according to the error GPTQ cannot compensate, with a per-input-channel column scaling whose power-of-two normalization keeps the scaled activations out of the FP16 subnormal range, so that the quantized model maps directly onto FIGFLEX's datapath with the same accuracy as on a GPU. On LLaMA-2-7B, FIGFLEX attains 5.85 and 8.77 WikiText-2 perplexity at 3- and 2-bit average precision, lower than GPTQ, AWQ, SliM-LLM, AMQ, and QuIP# (without fine-tuning), while delivering 1.56x and 1.42x higher whole-chip throughput per area on the linear GEMMs of a 3-bit mixed-precision model than AxCore and FIGLUT, respectively.

I. Introduction

LLM inference is dominated by linear GEMMs, but low-bit mixed-precision weights do not necessarily reduce their compute cost on existing hardware. FIGFLEX co-designs a precision-flexible accelerator and quantizer to make lower weight precision reduce computation as well as memory use.

A. LLM Computation

  • Linear layers multiply activations by static weights and dominate transformer parameter count and runtime, making their GEMMs the target for acceleration.The paper describes linear GEMMs as the primary obstacle to efficient LLM serving.Source: §I · p. 1, A. LLM Computation · p. 2

I. Introduction

  • Mixed-precision quantization assigns different bit budgets to weight groups, but existing storage savings do not automatically translate into cheaper computation.The paper identifies this mismatch as the opportunity for hardware to turn compression into faster linear GEMMs.Source: §I · p. 1
  • GPUs dequantize low-bit weights to FP16 before multiply-accumulate, while dedicated bit-parallel FP-INT accelerators support fixed precision and cannot increase throughput as weight bits fall.FIGLUT supports mixed precision, but its work grows with weight bit-width and its lookup table grows exponentially with the number of grouped activations.Source: §I · pp. 1–2
  • FIGFLEX obtains 5.85/8.77 WikiText-2 perplexity at 3-/2-bit average precision on LLaMA-2-7B, and on a 3-bit mixed-precision model achieves 1.56×/1.42× AxCore/FIGLUT whole-chip throughput per area.Source: §I · p. 2

A. LLM Computation

  • Figure 1 models linear GEMMs, attention, and memory separately, comparing GPU low-bit kernels with FIGFLEX at matched bit-width and bandwidth.FIGFLEX GEMM rates are based on measured compute density and rates, while attention and memory are modeled as for the GPU.Source: Figure 1 · p. 2

B. Weight-Only Quantization

Weight-only post-training quantization keeps activations in FP16 while storing weights as low-bit integers, but mixed-precision group widths challenge hardware built for one precision. Existing approaches trade flexibility against compute efficiency, motivating a bit-parallel datapath that scales with weight precision.

B. Weight-Only Quantization

  • Weight-only PTQ stores weights as low-bit integers while retaining FP16 activations, using asymmetric affine codes with a scale and zero-point per 128-input-channel group.GPTQ minimizes layerwise output error and propagates rounding error through the calibration Hessian so later columns compensate part of it.Source: B. Weight-Only Quantization · p. 2
  • Below 4 bits, mixed-precision methods vary weight-group widths, but their non-power-of-two precisions are not exploitable by fixed-precision hardware.SliM-LLM allocates by salience and AMQ searches per-layer allocations.Source: B. Weight-Only Quantization · p. 3

C. FP-INT GEMM Computation

  • GPU low-bit kernels dequantize weights to FP16, so arithmetic cost is independent of bit-width; FIGNA and AxCore also process each weight at one fixed width.GPU gains therefore saturate when GEMMs are compute-bound, while lower weight precision does not raise the fixed-width accelerators’ throughput.Source: C. FP-INT GEMM Computation · p. 3
  • FIGLUT scales throughput by processing one weight bit-plane per step, but its lookup table and selection logic grow exponentially with grouped activations.A b-bit weight still takes b reads, creating a different area and work trade-off from fixed-precision designs.Source: C. FP-INT GEMM Computation · p. 3

III. FIGFLEX Architecture

FIGFLEX uses a bit-parallel FP-INT processing element that turns multiplication and accumulation into a shared integer reduction. Radix-4 recoding lets the same datapath process multiple precisions, with lane fusion recovering throughput for INT3.

III. FIGFLEX Architecture

  • Figure 2 depicts the radix-4 datapath: per-lane 4:1 multiplexers feed one adder tree and an integer accumulator, with INT3 fused-lane wiring shown in green.Source: Figure 2 · p. 3

A. Design Principles

B. Radix-4 FP-INT Datapath

  • A radix-4 digit selects one of four aligned-significand multiples, and a shared adder tree and integer accumulator form products and their sum without a multiplier.The radix-4 recoding reduces partial products by grouping weight bits in pairs; the precomputed 3× multiple and shifts are generated at the row edge.Source: A. Design Principles · p. 3, B. Radix-4 FP-INT Datapath · p. 4
  • The PE consumes two weight bits per lane, sustaining 8, 16, and 32 MAC/cycle for INT4, INT2, and INT1, respectively, with K=16 lanes.Reconfiguring precision changes replicator routing while sharing the select lanes, adder tree, and accumulator.Source: B. Radix-4 FP-INT Datapath · p. 4
  • Naive INT3 uses two lanes per weight but leaves half of every second lane idle, limiting its rate to 8 MAC/cycle.This is no better than INT4 despite using fewer weight bits.Source: B. Radix-4 FP-INT Datapath · p. 4

C. INT3 Optimization

FIGFLEX fuses paired INT3 high bits into one shared radix-4 lane, reclaiming datapath capacity that would otherwise be idle. The weight aligner reorders the bits so the PE can consume the fused selectors contiguously.

C. INT3 Optimization

  • A shared 4:1 mux selects the paired high-bit contribution from 0, 4a0, 4a1, or 4(a0+a1).The inputs are formed using the lane’s existing adder; ×4 is wiring.Source: C. INT3 Optimization · p. 4
  • Two INT3 weights use three lanes instead of four, sustaining 2K/3 MAC/cycle with no idle bits, up from K/2.For K=16, the array reaches 10.67 MAC/cycle.Source: C. INT3 Optimization · p. 4
  • Across precisions, every lane uses the same 4:1 mux; only the replicator’s routing changes.The table describes per-lane radix-4 operations, with aligned activations and ×4 shifts implemented as wiring.Source: Table I · p. 4
  • The weight aligner places each pair’s two low digits before its concatenated MSBs, so every third lane receives a fused selector.The repacking is an exact bit reordering.Source: C. INT3 Optimization · p. 5, Figure 3 · p. 5

D. Full Accelerator

FIGFLEX combines shared row-edge units with an M×N radix-4 PE array and per-column output conversion. Its dataflow reorders channels by precision and keeps scaled activations stationary while weights stream across output channels.

D. Full Accelerator

  • Offline precision sorting splits each layer into uniform-precision GEMMs, while on-chip activations stay stationary as weights stream across output channels.A group’s activations are fetched once; tiles spanning two precisions run in the higher mode, keeping utilization above 99% on Llama-3.1-8B and above 93% for its smallest layers.Source: D. Full Accelerator · p. 5, Figure 5 · p. 5
  • An M×N array of radix-4 PEs is served by shared per-row edge units, with per-column converters and an epilogue producing the FP output.The edge units include an exponent aligner, radix-4 replicator, and weight aligner.Source: Figure 4 · p. 5, D. Full Accelerator · p. 5
  • Column scales are folded into activations before the GEMM, and the epilogue applies sign, per-channel scale, and per-group affine to the scalar accumulator output.Weights are never reconstructed to FP16, so no dequantization datapath is required.Source: D. Full Accelerator · p. 5

IV. Proposed Mixed-Precision Quantization

The quantizer builds on GPTQ with per-input-channel column scaling and compensation-aware mixed-precision assignment. Column scaling normalizes weight columns before quantization and returns their factors through activations at inference.

A. Overview

  • FIGFLEX adds column scaling and compensation-aware assignment over GPTQ, assigning precision separately to each group of an output channel.Groups contain 128 input channels; precision choices are W4, W3, W2, and W1.Source: A. Overview · pp. 5–6

B. Column Scaling

  • Each input-channel weight column is normalized by its magnitude before quantization, with the scale folded back into activations at inference.A power-of-two layer rescaling places the largest calibration product |ak c′k| at 2^12 to avoid FP16 subnormal-range underflow; on Llama-3.1-8B at 2 bits, removing column scaling raises perplexity from 13.77 to 16.48.Source: B. Column Scaling · p. 6

C. Compensation-Aware Assignment

Compensation-aware assignment allocates bits according to the GPTQ error that remains after compensation, rather than a pre-quantization salience score. It uses a global channel share and an additional GPTQ pass per layer.

C. Compensation-Aware Assignment

  • Column scaling flattens group sensitivity from 4.97× to 1.66×, and salience-based SBA is no better than uniform 3-bit on Llama-3.1-8B, at 7.237 vs. 7.227.The method instead ranks channels by the error remaining after GPTQ compensation.Source: C. Compensation-Aware Assignment · p. 6
  • A first GPTQ pass estimates each channel’s share of uncompensated loss, and assignment promotes channels with the largest scores while demoting those with the smallest.The second GPTQ pass quantizes the original weights at the assigned widths.Source: C. Compensation-Aware Assignment · p. 6
  • The assignment moves a global share of channels rather than searching per layer, costing one extra GPTQ pass per layer.The global shares are 1% for F3 and 20% for F2.Source: C. Compensation-Aware Assignment · p. 7

D. Storage Overhead

FIGFLEX’s storage overhead includes compact per-group metadata and per-input-channel scaling. The evaluation compares quantization and hardware baselines under specified model, synthesis, and energy-measurement settings.

D. Storage Overhead

  • The reported 3.0- and 2.0-bit averages count weight codes only; tags add 0.016 bits per weight, while column scales add below 0.004 bits per weight for N ≥ 4096.Each group stores one FP16 scale, an integer zero point, and a 2-bit precision tag; the per-input-channel vector c stores one FP16 value per channel.Source: D. Storage Overhead · p. 7

A. Experimental Setup

  • Six open models are evaluated on WikiText-2 and C4 perplexity and four zero-shot accuracy benchmarks, with decoder linear layers quantized and no method fine-tuned.The evaluation uses F3 and F2 at 3.00 and 2.00 average bits, respectively; embeddings, the LM head, and activations remain FP16.Source: A. Experimental Setup · p. 7
  • The hardware comparison includes FIGNA, AxCore, and FIGLUT, reimplemented and synthesized in the authors’ flow alongside conventional floating-point FMA and dot-product engines.AxCore uses approximate logarithmic-domain multiplication, while FIGLUT replaces the multiplier with a table read.Source: A. Experimental Setup · p. 7
  • Hardware designs are synthesized in NanGate 15 nm at a common 1 GHz clock and adjusted to the same maximum-performance target on a 128×128 array.The setup re-synthesizes designs for FP32, FP16, or BF16 activations and INT4, INT3, or INT2 weights; SRAM and off-chip memory are included in energy accounting.Source: A. Experimental Setup · pp. 7–8

B. Numerical Fidelity

FIGFLEX preserves model quality through its datapath while maintaining numerical fidelity against GPU and tensor-core comparisons. Its model-quality results are strongest among affine methods at 3 bits and across most models at 2 bits.

B. Numerical Fidelity

  • F3 and F2 Llama-3.1-8B WikiText-2 perplexity matches GPU to the third decimal: 8.2511 vs. 8.2508 and 15.3005 vs. 15.3004.The RTL integer datapath matched bit for bit on all 10.0 million group results tested.Source: B. Numerical Fidelity · p. 8
  • For FP16, F3 mean relative error is 2.0 × 10^-3 vs. 2.5 × 10^-3 for the tensor core; radix-4 recoding and integer accumulation are lossless.The only rounding comes from shared-exponent alignment; Figure 7 compares errors over 21 projections and GPU/datapath perplexity.Source: B. Numerical Fidelity · p. 8, Figure 7 · p. 9

C. Model Quality

  • At 2 bits, FIGFLEX averages 21% below QuIP# and is best overall on five of six models; at 3 bits, it has the lowest WikiText-2 perplexity among affine methods on all six.At 3 bits, it is lowest overall on four models; QuIP# leads on the other two by at most 0.07.Source: C. Model Quality · p. 8
  • At 3 bits, FIGFLEX averages 69.7% zero-shot accuracy versus 69.2% for AWQ and 67.5% for GPTQ, within 0.7 points of QuIP#.At 2 bits, it leads GPTQ and SliM-LLM by 14.9 and 7.0 points, respectively, while trailing AMQ by 2.6 points and QuIP# by 9.1.Source: C. Model Quality · p. 9

D. Quantizer Ablation

The compensation-aware quantizer improves low-bit model quality, while FIGFLEX’s radix-4 datapath turns lower precision into higher GEMM throughput and compute density.

D. Quantizer Ablation

  • At 2 bits, CA-Mixed lowers perplexity from 98.94 for GPTQ to 16.48, and column scaling lowers it further to 13.77.The tuned uniform model reaches 29.28, while SBA-Mixed reaches 18.84 at 2 bits.Source: D. Quantizer Ablation · p. 9
  • At 2 bits, F2 achieves 2.1× lower perplexity than uniform W2 with a 1% tile-packing loss; equal average bits yield equal throughput.The reported prefill times are 453.2 ms for F2 and 448.8 ms for uniform W2.Source: D. Quantizer Ablation · p. 9

E. Hardware Efficiency

  • Radix-4 cuts PE area by 24% and area per MAC by 23%; INT3 fusion raises throughput from 8 to 10.67 MAC/cycle while reducing area per INT3 MAC from 110.8 to 84.1 µm2.The INT3 fusion adds 1.3% area.Source: E. Hardware Efficiency · p. 9
  • At F3 and F2, FIGFLEX is 1.23× and 1.64× faster than fixed-precision designs; FIGLUT matches its runtime but uses 1.36× the area and 1.30× the power.At INT4, all designs take the same request time under the iso-TOPS configuration.Source: E. Hardware Efficiency · p. 10
  • At INT3, FIGFLEX reaches 1.97× FIGNA and 1.56× AxCore compute density; FIGLUT trails FIGFLEX by 1.42× at INT3 and INT2.The reported FIGFLEX advantage over FIGNA and AxCore grows at INT3, while its advantage over precision-scalable FIGLUT reflects per-MAC cost.Source: E. Hardware Efficiency · pp. 10–11, Figure 8 · p. 10
  • FIGFLEX consumes the least energy at every operating point and surpasses FIGLUT in energy efficiency by an average of 25%.The energy comparison covers uniform INT4 and mixed-precision INT3 and INT2 runs of Qwen3-32B.Source: E. Hardware Efficiency · p. 11, Figure 10 · p. 11
  • At batch 128, FIGFLEX shortens modeled requests by 2.2–2.9× at F3 and 2.9–3.7× at F2; at batch 1, gains fall to 1.02–1.16× and 1.04–1.27×.The batch-1 decode case is memory-bound, with both systems reading the same bytes.Source: E. Hardware Efficiency · p. 11

A. Precision-Scalable DNN Accelerators

FIGFLEX combines a bit-parallel datapath that scales with weight precision and an allocator matched to the precisions it executes. The conclusion reports 5.85 and 8.77 WikiText-2 perplexity at 3- and 2-bit average precision, alongside higher throughput per area than AxCore and FIGLUT on 3-bit linear GEMMs.

A. Precision-Scalable DNN Accelerators

  • FIGFLEX is bit-parallel like BitBlade, but scales only the weight operand and uses one integer adder tree after block-floating-point pre-alignment.Bit-serial designs process one bit-plane per cycle, so their latency grows with precision.Source: A. Precision-Scalable DNN Accelerators · p. 12

B. Beyond Weight-Only Quantization

  • Codebook quantizers such as QuIP# and sparse-outlier formats such as SpQR [8] reach strong quality at 2–3 bits, but require codebook or index decoding.That decoding reintroduces dequantization or requires a LUT datapath; activation and KV-cache quantization address complementary operators or phases.Source: B. Beyond Weight-Only Quantization · p. 12

C. Algorithm–Hardware Co-Design for Quantized LLMs

  • FIGFLEX closes the co-design loop by matching its allocator’s {W4, W3, W2, W1} precision set to the array’s supported formats and 32/b MAC/cycle rate.Earlier mixed-precision PTQ methods select widths for model quality, while precision-flexible accelerators were evaluated on externally produced models.Source: C. Algorithm–Hardware Co-Design for… · p. 12

VII. Conclusion

  • On LLaMA-2-7B, FIGFLEX attains 5.85 and 8.77 WikiText-2 perplexity at 3- and 2-bit average precision, the lowest among the compared affine post-training quantizers.On 3-bit linear GEMMs, it delivers 1.56× and 1.42× the whole-chip throughput per area of AxCore and FIGLUT, respectively.Source: §VII · p. 12
Loading 2610.30480v1…