PaperlayerView exact revision on arXiv

Source-linked AI summary

AlignQuant: Tile-Aligned Mixed-Precision Quantization for Efficient LLM Generation

Hanzhi Zhang, Qiao Zhang, Qinglei Cao, Heng Fan, Yan Huang, Kewei Sha, Yunhe Feng

arXiv:2610.07457v1cs.LGcs.AIcs.CL

TL;DR

Fine-grained mixed-precision quantization faces a mismatch between local precision choices and regular GPU storage and computation units. AlignQuant uses shared two-dimensional weight tiles for allocation, storage, and execution; across four LLMs, it reports up to 2.50× inference speedups over BF16 while preserving model quality.

  • Problem

    Fine-grained mixed-precision quantization must preserve local precision choices while accommodating regular GPU storage and computation units.

    Source: §1 · pp. 1–2

  • Method

    AlignQuant uses GPU-compatible two-dimensional weight tiles as a shared unit for local precision allocation, compact storage, and phase-specialized execution.

    Source: §1 · p. 2

  • Results

    Across four LLMs, AlignQuant achieves maximum 2.50× inference speedups over BF16 on three GPUs while preserving model quality.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Fine-grained precision choices can coexist with regular GPU computation when the precision unit supports sensitivity evaluation and efficient execution.

    Source: §5 · p. 9

    Because all tiles contain the same number of weights, each 8-to-4-bit replacement saves the same storage, so the budget sets the number of 4-bit tiles.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Fine-grained mixed-precision quantization promises efficient large language model inference, but local precision choices can conflict with regular GPU storage and computation units. This precision-boundary mismatch limits the translation of compression into practical acceleration. We introduce AlignQuant, a post-training quantization method that uses GPU-compatible two-dimensional weight tiles as the common unit of precision allocation, compact storage, and execution. This shared partition lets precision follow sensitivity within output channels. Joint prefill/decode calibration scores precision reductions using projection-output perturbations weighted by language-model loss gradients under quantized activations. Phase-normalized scores prioritize higher precision for tiles important to either phase under a model-wide weight-storage budget. Each tile stores one selected representation, while phase-specialized kernels reuse the packed model and expand lower-bit weights for INT8 computation with 8-bit activations. Across four LLMs spanning 3B to 14B parameters, AlignQuant achieves up to $2.50\times$ generation speedup over BF16 while preserving model quality. Evaluations further cover three GPUs and contexts up to 64K tokens. These results show that local precision flexibility and regular GPU execution can coexist through a shared tile unit. The implementation is available at https://github.com/HanzhiZhang-Ulrica/AlignQuant.

1 INTRODUCTION

AlignQuant addresses the conflict between fine-grained precision choices and regular GPU storage and computation units. It uses shared weight tiles for precision allocation, compact storage, and execution, and reports faster inference while preserving model quality.

1 INTRODUCTION

  • Fine-grained precision boundaries can conflict with regular GPU storage and computation units, making deployment depend on how mixed-bit weights are packed, loaded, and converted.The introduction frames the problem as preserving sensitivity-driven local precision choices in compact storage and direct GPU execution.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • GPU-compatible two-dimensional weight tiles provide a shared unit for local precision selection, storage, and execution, allowing precision to vary within output channels.The paper reports localized sensitivity within output channels and improved WikiText-2 perplexity over an output-row control at a matched 4-bit tile fraction.Source: §1 · p. 2
  • Across four LLMs, AlignQuant achieves maximum 2.50× inference speedups over BF16 on three GPUs while preserving model quality.The evaluated models are Llama-3.2-3B, Qwen3-4B, Ministral-3-8B, and Qwen3-14B.Source: §1 · p. 2
  • Joint loss-aware calibration scores 8-to-4-bit substitutions under 8-bit activations and allocates precision across prefill and decode under one model-wide storage budget.Each tile stores one selected representation; phase-specialized kernels expand 4-bit weights on-chip so both weight formats share INT8 arithmetic.Source: §1 · p. 2

2 RELATED WORK

Prior work varies the granularity of precision allocation, the representation and execution of heterogeneous weights, and transformations used to reduce quantization error. These choices jointly shape sensitivity handling and GPU efficiency.

2 RELATED WORK

  • Allocation methods range from layer-wise and channel-, feature-, or group-wise units to hardware-aligned blocks and phase-adaptive precision, trading local sensitivity choices against structured partitions and phase sharing.The review notes that allocation depends on spatial granularity, sensitivity evidence, and precision sharing across inference phases.Source: §2 · p. 3
  • Low-bit systems co-design formats, packing, data movement, and kernels, while executable mixed-precision systems organize heterogeneous weights through partitions, layouts, indexing, or specialized arithmetic paths.Representation-centric approaches use outliers, residual subspaces, binary codes, or fractional-bit quantizers, often with corresponding reconstruction or access mechanisms.Source: §2 · p. 3
  • Scaling, rotations, permutations, affine transforms, and runtime smoothing reshape distributions to reduce quantization error, while spatial bit-width placement remains a separate decision in several approaches.The review concludes that executable mixed precision also requires compatible transformation, allocation, and execution partitions.Source: §2 · p. 3

3 AlignQuant

AlignQuant uses a shared two-dimensional weight-tile partition for precision allocation, compact storage, and GPU execution. Joint prefill/decode scoring assigns precision under a model-wide budget, and phase-specialized kernels execute the same packed model.

3.1 TILE-LEVEL PRECISION ASSIGNMENT

  • A model-wide budget assigns 4-bit weights to the lowest-scoring equal-sized 2D tiles, retaining 8-bit weights for tiles with high relative sensitivity in either phase.The target 4-bit fraction sets how many tiles are reduced; scores determine which ones.Source: §3.1 · pp. 4–5
  • Tile-aligned U/V transforms produce 4-bit and 8-bit candidates, and calibration uses the same 8-bit activation quantizer as inference.Comparing reconstructed candidates under deployed activation quantization makes the candidate comparison reflect inference inputs.Source: §3.1 · p. 4
  • Each 8-to-4-bit substitution is scored by its activation-dependent projection perturbation weighted by loss gradients, allowing distinct scores for tiles sharing output features.Tile sensitivity varies within query-projection matrices and across decoder depths.Source: §3.1 · p. 5, Figure 3 · p. 5

3.2 COMPACT MIXED-PRECISION REPRESENTATION

  • Packing stores one selected integer representation and reconstruction scale per tile, with 4-bit tiles using half the integer-weight storage of equally sized 8-bit tiles.The runtime locates each selected tile through separate contiguous precision buffers and a precision label.Source: §3.2 · p. 6

3.3 DIRECT PREFILL/DECODE EXECUTION

  • Phase-specialized kernels use Tensor Core INT8 multiplication for prefill and DP4A dot products for decode, expanding packed 4-bit weights into INT8 operands on chip.Long prefill can reuse a temporary INT8 buffer; the phase transition reuses the same precision map and selected weights.Source: §3.3 · p. 6
  • The shared tile design preserves local precision flexibility while making each 4-bit choice reduce stored weight bytes.Joint calibration and phase-specific weight preparation support generation with one compact model.Source: §3 · p. 3, §3.3 · p. 7

4 EXPERIMENTS

The experiments evaluate AlignQuant’s inference efficiency, general-task quality, and long-context performance against BF16 and quantization baselines. Results report faster generation alongside strong quality, including at longer contexts.

4.1 EXPERIMENTAL SETUP

  • The setup evaluates perplexity, reasoning and understanding benchmarks, LV-Eval QA at 16K–64K, and LongEval line-retrieval accuracy; primary measurements use an A100.Source: §4.1 · p. 7

4.2 INFERENCE EFFICIENCY

  • 132.28 tokens/s, 2.27× BF16 and 2.75× GPTQ throughput on Llama-3.2-3B; AlignQuant leads displayed throughput comparisons across Figure 5 request shapes.Source: §4.2 · p. 7, Figure 5 · p. 8
  • Figure 5 reports generation-throughput speedup relative to BF16 on Llama-3.2-3B across prompt lengths of 128, 2K, 8K, and 16K tokens and batch sizes of 1, 2, 4, and 8.Source: Figure 5 · p. 8

4.3 GENERAL QUALITY

  • AlignQuant achieves the lowest perplexity and highest or tied-highest accuracy among quantized configurations on general language tasks.Source: §4.3 · p. 8
  • 65.66% on LAMBADA and 71.22% on WinoGrande exceed structured and random placements, supporting local precision allocation within output channels.Source: §4.3 · p. 8

4.4 LONG-CONTEXT EVALUATION

  • At 64K, AlignQuant scores 48.06 on LooGLE-SD versus 48.81 for BF16, and 24.81 on LooGLE-CR versus 22.42 for reduction-column and 20.57 for output-row allocation.Source: §4.4 · pp. 8–9
  • At the longest LongEval input, AlignQuant matches BF16 at 48%, versus 40% for MixLLM and 31% for AWQ.Source: Figure 6 · p. 9, §4.4 · p. 9
  • At the longest retrieval inputs, AlignQuant’s 48% accuracy exceeds structured placements at 33%–39% and random placement at 25%.Source: §4.4 · p. 9

5 CONCLUSION

AlignQuant’s conclusion is that fine-grained precision choices can coexist with regular GPU computation when precision units support sensitivity evaluation, compact storage, and execution. The appendices detail its transformations, quantization candidates, calibration observations, and precision-map construction.

5 CONCLUSION

  • The paper concludes that tile units can align local output sensitivity with compact storage and regular GPU computation while preserving quality and boosting inference efficiency.Source: §5 · p. 9

A NOTATION AND READING GUIDE

  • Within an output-tile row, tiles share output features but span input columns; scores combine their shared output-gradient slice with tile-specific input perturbations.Source: A NOTATION AND READING GUIDE · p. 13

B.1 TILE-ALIGNED TRANSFORMATIONS

  • Block-diagonal U and V transforms redistribute tile values while preserving the floating-point projection after transformed inputs and weights are reconstructed.Source: §B.1 · p. 13

B.2 WEIGHT CANDIDATES

  • Each tile has 4-bit and 8-bit candidates, zero point zero, nearest-even rounding, and a separately fitted scale; packing retains the selected integer values and scale.Source: §B.2 · pp. 13–14

B.3 ACTIVATION QUANTIZATION AND PADDING

  • Activation quantization uses INT8 values with FP32 scales in groups of 64 features and row groups sized by M; padding is removed from the output afterward.Source: §B.3 · p. 14

B.4 TRANSFORM SELECTION

  • Transform selection searches eight seeds, refines input then output transforms in coordinate sweeps, and passes the resulting transforms to W4/W8 scoring under A8 inputs.Source: §B.4 · p. 14

C.1 FORWARD VALUES AND GRADIENT OBSERVATIONS

  • Calibration keeps model parameters fixed, records projection outputs and gradients, scores continuation targets in a causal prefill pass, and scores up to four sequential decode steps.Source: §C.1 · p. 14, §C.2 · p. 14

C.3 COMPUTING THE PRECISION MAP

  • For each input-tile column, the scorer computes tile scores by projection and phase, combines them for model-wide ranking, then selects and packs the target number of 4-bit tiles.Source: §C.3 · p. 15

D DERIVATIONS AND PROPERTIES OF THE ALLOCATION SCORE

AlignQuant scores local tile substitutions by their projection-output effects and loss-gradient importance, then selects a fixed-budget precision map that accounts for both prefill and decode. That map is stored and executed through tile-aligned paths, with shared input preparation for compatible projections.

D.1 LOCAL OUTPUT PERTURBATION / D.2 LOSS-WEIGHTED QUADRATIC FORM

  • A tile substitution combines weight errors through the input, while loss-gradient weighting evaluates the resulting projection-output changes in output-coordinate space.The calibration score uses captured states and the stated gradient rule; downstream quality is evaluated after assembling all selected substitutions.Source: §D.1 · p. 15, §D.2 · p. 15

D.3 PHASE NORMALIZATION

  • The joint score uses phase-normalized tile importance to rank lower-precision substitutions, so a fixed budget prioritizes tiles important to either prefill or decode.The selected higher-precision tiles are those outside the lowest-scoring fraction.Source: §D.3 · p. 15

D.4 FIXED-BUDGET SELECTION

  • For equal-area tiles, fixed-budget selection assigns W4 to the lowest-scoring k tiles, where k is the rounded target fraction of all tiles.Equal-area substitutions have equal integer-storage savings; endpoint budgets select no tiles or all tiles.Source: §D.4 · p. 16

E.1 LOCATING AND UNPACKING A TILE / E.2 STORED BYTES

  • Packed tiles retain their selected W4 or W8 format: precision labels choose the corresponding buffer, tiles are stored in row order, and weight scales use the same index.For 64 × 64 tiles, the W4 and W8 buffers have shapes [T4, 64, 32] and [T8, 64, 64], respectively.Source: §E.1 · p. 16, §E.2 · p. 16

E.3 OPERAND PREPARATION AND SCHEDULING

  • Phase-specialized kernels expand W4 weights on chip for INT8 computation; long prefill can prepare a temporary INT8 buffer once per projection call and reuse it across activation-row groups.Both paths retain selected integer values and scales and use separately prepared 8-bit activations.Source: §E.3 · pp. 17–18, Figure 7 · p. 17

E.4 SHARED PROJECTION INPUTS

  • Compatible query/key/value or gate/up projections share one transformed, quantized input, while each retains its own selected weights, scales, output transform, and output slice.Their packed weight rows and output-transform metadata are assembled during model setup.Source: §E.4 · p. 18, Figure 8 · p. 18

F.1 ACTIVATION PRECISION / F.2 PRECISION GRANULARITY AND EXECUTION

  • The method uses 8-bit activations; A4 would require recalibrating candidate scores and preparing reconstructed inputs for the chosen arithmetic.The weight-tile precision choice remains independent of activation groups and thread-block execution geometry.Source: §F.1 · p. 19, §F.2 · p. 19

G.2 CALIBRATION SETTINGS / G.3 GENERAL QUALITY EVALUATION / G.4 LONG-CONTEXT EVALUATION

  • Calibration uses ordered Multi-News document-summary pairs for prefill and deterministic matched-prefix windows for decode, with the default model-wide allocation assigning W4 to the lowest-scoring 60% of tiles.The evaluation protocols specify word-level WikiText-2 perplexity, six-task equal-weight General Avg accuracy, and long-context benchmarks including LV-Eval and LongEval.Source: §G.2 · pp. 19–20, §G.3 · p. 20, §G.4 · p. 20

G.5 SYSTEM EVALUATION

The system evaluation measures generation performance under matched runtime settings and reports memory using packed-model and runtime-allocation accounting. The accompanying protocols vary workload dimensions, calibration conditions, and precision budgets.

G.5 SYSTEM EVALUATION

  • Generation measurements use batch size 1, a 128-token prompt, and 512 generated tokens, with median latency, throughput, and energy reported across timed repetitions.Measurements include online operand preparation and projection computation; headline speedups are matched-workload BF16-to-AlignQuant TPOT ratios.Source: §G.5 · p. 20
  • The system sweep varies prompt length, output length, and batch size across 80 settings, while a separate peak-memory sweep uses prompt lengths from 128 to 16K tokens.The reported comparison includes AlignQuant and several floating-point and quantized methods.Source: §G.5 · p. 20

G.6 ABLATION PROTOCOLS / G.7 STORAGE AND MEMORY ACCOUNTING

  • Ablations vary tile size, calibration-document count, W4 budget, and runtime controls; packed BPW and full-model storage are calculated separately from runtime GPU memory.Memory accounting includes activations, key/value states, and temporary weights according to their runtime lifetimes.Source: §G.6 · p. 20, §G.7 · p. 20

H.1 GENERAL QUALITY RESULTS

Across Qwen3-4B, Ministral-3-8B, and Qwen3-14B, selective W8 retention improves reported task scores over W4A8 while preserving predictive behavior near BF16. Matched-budget comparisons also favor local tile placement over structured and random controls.

H.1 GENERAL QUALITY RESULTS

  • Across three additional models, selective W8 retention improves reported task scores over W4A8 while preserving predictive behavior near BF16.The evaluations cover both downstream accuracy and language-model perplexity.Source: §H.1 · p. 21, Table 8 · p. 22
  • Qwen3-4B LAMBADA accuracy reaches 63.38%, alongside 63.40% for BF16 and 58.14% for W4A8.These values are reported for the same task and model.Source: §H.1 · p. 21
  • Qwen3-4B reaches 13.15 perplexity versus 13.32 to 13.88 for structured and random placements; Qwen3-14B LAMBADA accuracy is 69.65% versus 67.55% to 67.79% for structured controls and 66.45% for random placement.The comparisons hold the W4 fraction fixed.Source: §H.1 · p. 21, Table 9 · p. 22

H.2 LV-EVAL TASK RESULTS

LV-Eval results show model- and task-specific behavior across context lengths, while allocation and tile-size analyses examine the precision maps behind those outcomes. The W4-budget sweep reports lower packed storage with similar throughput across tested mixed configurations.

H.2 LV-EVAL TASK RESULTS

  • At 64K, Qwen3-14B LooGLE-SD scores 17.86 with AlignQuant versus 8.55 with W4A8; on English multi-field QA, its score is 14.28 versus 14.66 for BF16 and 5.89 for W4A8.Qwen3-4B CMRC scores 24.95, 25.30, and 24.74 at 16K, 32K, and 64K, with the final value matching BF16.Source: §H.2 · p. 22, Table 11 · p. 23, Table 13 · p. 25
  • With the same W8 capacity, Qwen3-14B LooGLE-SD scores 21.51 at 16K and 17.86 at 64K, versus 21.03 and 10.11 for the strongest structured control.On Ministral-3-8B LooGLE-CR at 64K, AlignQuant scores 26.77 versus 21.45 to 23.37 for structured and random allocations.Source: §H.2 · p. 22, Table 12 · p. 24, Table 13 · p. 25

H.3.1 ALLOCATION STRUCTURE AND SENSITIVITY LOCALIZATION

  • Mixed W4/W8 assignments occur in 59.81% of Llama-3.2-3B output-tile rows and 87.37% to 90.17% of rows in the other models.These rows contain input regions contributing to the same output-feature group but assigned different precisions.Source: §H.3.1 · p. 23, Table 14 · p. 25

H.3.2 TILE-SIZE ABLATION

  • 64 × 64 tiles give the lowest prefill and decode NLL in all six tested model-phase comparisons; 32 × 32 uses four times as many scales and labels.The 128 × 128 partition increases NLL in the tested settings.Source: §H.3.2 · p. 24

H.3.3 SELECTED-WEIGHT STORAGE AND W4 BUDGET

  • Across mixed configurations from 40% to 80% W4, packed storage decreases from 6.41 to 4.81 bits per weight and resident GPU memory from 2.98 to 2.45 GiB.Throughput remains between 130.51 and 134.83 tokens/s; the default 60% configuration reaches 132.28 tokens/s at 5.61 packed bits per weight.Source: §H.3.3 · p. 24

H.3.4 CALIBRATION-DATA SENSITIVITY

The calibration-data sensitivity study finds similar downstream quality across tested calibration-set sizes and sampling seeds. It varies the evidence guiding tile placement while keeping the representation and storage objective fixed.

H.3.4 CALIBRATION-DATA SENSITIVITY

  • Task-quality scores remain in narrow ranges across 8, 16, 24, and 48 calibration documents, with sample standard deviations of 0.03–0.11 percentage points across seeds.Each size uses three seeds; comparable quality is retained across a sixfold range of document counts.Source: §H.3.4 · p. 25
  • Table 15 reports a tile-size ablation across three models using held-out prefill and decode negative log-likelihood.The caption identifies the best and worst values by model but supplies no values for comparison.Source: Table 15 · p. 26
  • The sampling sweep changes the evidence guiding tile placement while retaining the same representation and storage objective.Each sampled record contributes prompt-processing and token-generation observations; phase normalization converts accumulated scores to relative tile importance under a common 4-bit target.Source: §H.3.4 · p. 26

H.4.1 WORKLOAD SCALING

The workload-scaling results assess AlignQuant across prompt lengths, output lengths, and batch sizes. The same selected W4/W8 model improves throughput, per-token latency, and energy efficiency, with gains varying by workload.

H.4.1 WORKLOAD SCALING

  • The throughput advantage extends across prompt lengths, output lengths, and batch sizes.Figures 9 and 10 examine the same workload grid through per-token latency and energy efficiency.Source: §H.4.1 · p. 26
  • AlignQuant uses less energy per generated token than BF16 across the displayed workloads, with gains generally increasing with continuation length for 8K and 16K prompts.Longer continuations spread prompt processing and weight preparation over more generated tokens.Source: §H.4.1 · p. 26
  • The same selected W4/W8 model improves aggregate throughput, per-token latency, and energy efficiency across the tested request shapes.The magnitude of each gain changes with workload composition.Source: §H.4.1 · p. 26
  • TPOT speedup exceeds 2× for several batch-one workloads and also extends to 16K-input workloads at batch sizes 4 and 8.Figure 9 reports TPOT speedup relative to BF16 over batch sizes 1, 2, 4, and 8 and prompt lengths 128, 2K, 8K, and 16K tokens.Source: §H.4.1 · p. 26, Figure 9 · p. 27

H.4.2 CROSS-HARDWARE AND CROSS-MODEL RESULTS

Across model sizes and GPUs, AlignQuant reports more efficient generation while preserving model quality. The evaluations also show lower peak memory across prompt lengths.

H.4.2 CROSS-HARDWARE AND CROSS-MODEL RESULTS

  • The common tile representation lets each projection store selected values and scales while execution reuses integer computation across the evaluated projection geometries.The execution organization distributes work over output tiles and input-column groups; selected W4 values are expanded for common INT8 arithmetic.Source: §H.4.2 · pp. 27–28
  • 167.49 tokens/s on H100 NVL and 180.64 tokens/s on RTX 4090 exceed the respective BF16 throughputs of 81.54 and 108.05 tokens/s.Both GPU measurements also report lower energy per generated token and a resident footprint of approximately 2.7 GiB.Source: §H.4.2 · p. 27, Table 18 · p. 28, Table 19 · p. 29
  • 1.53× to 2.50× TPOT speedups over BF16 are reported across four models; on Qwen3-14B, TPOT falls from 30.29 to 19.85 ms and energy per 1,000 generated tokens from 6972.05 to 4354.57 J.All four models were tested on one A100 with batch size 1, a 128-token prompt, and 512 generated tokens.Source: §H.4.2 · p. 27, Table 20 · p. 29
  • At 16K prompt tokens, AlignQuant uses 4.44 GiB peak VRAM versus 8.88 GiB for BF16, half the BF16 peak.Across 128 to 16K prompt tokens, peak allocation rises from 2.72 to 4.44 GiB for AlignQuant and from 6.09 to 8.88 GiB for BF16.Source: §H.4.2 · p. 28, Table 21 · p. 29

H.5 RUNTIME-CONTROL RESULTS

Runtime controls isolate contributions from submission, input reuse, and decode scheduling while holding the selected weights fixed. Shared activation preparation and CUDA Graph execution improve reported generation metrics.

H.5 RUNTIME-CONTROL RESULTS

  • CUDA Graph execution lowers TPOT from 21.07 to 7.54 ms and raises throughput from 47.45 to 132.28 tokens/s, with peak GPU memory nearly unchanged.The comparison changes computation submission while holding the selected numerical model fixed.Source: Table 22 · p. 30, §H.5 · p. 30
  • Shared activation preparation reduces TTFT from 29.80 to 19.67 ms and TPOT from 10.42 to 7.54 ms relative to separate preparation.A common transformed 8-bit input is prepared for compatible query/key/value or gate/up projections.Source: Table 22 · p. 30, §H.5 · p. 30
  • Adaptive decode splitting lowers TPOT from 8.54 to 7.54 ms while TTFT remains approximately 19.6 ms.The change is concentrated in the generation schedule, which distributes reduction work across output features and input-column groups.Source: §H.5 · pp. 29–30, Table 22 · p. 30
  • The controls identify runtime improvements from reduction scheduling, input reuse across projections, and submission of repeated model computation.These execution choices operate on values selected by the same calibration and packing procedure.Source: §H.5 · p. 30

Related papers

Loading 2610.07457v1…