RPTQ: Reorder-based Post-training Quantization for Large Language Models

1 Introduction

• Large LLM deployment strains memory, and distributing model chunks across devices entails data-transfer bandwidth and energy costs.Source: §1 · p. 1
• Activation channels vary substantially in value range, so applying one quantization parameter across channels can cause large errors even when their outlier magnitudes match.Source: §1 · p. 2
• RPTQ clusters activation channels with similar value ranges for shared-parameter quantization and fuses reordering into layer norm and linear-layer weights.Source: §1 · p. 2
• Experiments report that clustering channels into a small number of clusters reduces quantization errors and improves quantized-LLM accuracy.Source: §1 · p. 2
• OPT-175b memory usage falls 73% with perplexity loss below 0.5 or 80% with loss below 1.5 across sequence lengths and batch sizes.Source: §1 · p. 3

2.1 Large Language Model

• LLM memory demands have motivated compression, distributed computing, and computational-graph optimization as deployment strategies.Source: §2.1 · p. 3

2.2 Quantization

• PTQ quantizes pretrained models without additional training, whereas QAT trains with quantization constraints and can require substantial computation.Source: §2.2 · p. 3
• Prior LLM methods include mixed-precision outlier handling [11], per-channel scaling [34], and second-order weight quantization [15], but reviewed methods generally quantize activations only to 8 bits.Source: §2.2 · p. 3
• PTQ-SL [37] reorders weights for convolutional-network weight quantization, while PGQ [3] groups embedding dimensions using dynamic ranges and uniformly divided groups.Source: §2.2 · p. 3

3.1 Post-training Quantization

• Uniform quantization maps float values to k-bit integers, with scale and zero point used in quantization and de-quantization.Source: §3.1 · p. 4
• Static quantization computes activation parameters before deployment and reuses them at runtime; the paper focuses on this setting.Source: §3.1 · p. 4
• Calibration samples capture likely inference-time activation distributions, and Min-Max parameter selection uses each tensor’s maximum and minimum values.Source: §3.1 · p. 4

3.2 Challenges in Activation Quantization

• Per-tensor quantization can incur large errors when channels span different ranges, because one shared range may underrepresent small values or truncate outliers.Source: §3.2 · p. 4
• Prior approaches include mixed-precision outlier treatment, per-channel scaling, and shared parameters for adjacent activation values, with the latter adding parameter computation and storage overhead.Source: §3.2 · pp. 4–5
• Figure 2 presents methods for addressing activation quantization challenges.Source: Figure 2 · p. 5

4.1 Clustering and Reordering of Channels

• K-means groups channels by per-channel minimum and maximum values, then reorders channels from each cluster together for quantization.The reordered activations are formed by concatenating the cluster index sets.Source: §4.1 · pp. 5–6
• Cluster-level quantization parameters address channel-range differences and reduce quantization error while requiring fewer parameters than per-channel or per-vector approaches.Channels with similar value ranges share a scale and zero point.Source: §4.1 · p. 6

4.2 Avoid Explicit Reordering and Misalignment

• Explicit runtime reordering moves channel data between memory locations, increasing inference latency and memory overhead; RPTQ fuses the operation into other operations.Layer normalization writes results to reordered addresses using the reorder index.Source: §4.2 · p. 6
• Linear-layer weights are reordered by input and output channel indices so the layer accepts reordered inputs and produces reordered outputs without inference-time weight-reordering overhead.The weight reordering can be completed before deployment.Source: §4.2 · pp. 6–7
• Residual connections require the out-projection and final-linear outputs to retain the input channel order, while query and key activations share a reorder index for QK-MatMul.The shared query-key index is based on clustering their combined per-channel extrema.Source: §4.2 · p. 7

5.1 Settings

• The evaluation uses GPTQ for weight quantization, static quantization for weights and input activations, and 256 calibration samples from three datasets.OPT models are evaluated with perplexity, zero-shot tasks, memory consumption, and a cluster-count ablation; output activations of layer norm and softmax use 8-bit precision.Source: §5.1 · p. 7, §5.2 · pp. 7–8

5.2 Results on LLM

• For OPT-175b, W4A8 achieves perplexity loss below 0.5, while W4A4 achieves perplexity loss below 3.The reported comparisons cover perplexity and zero-shot accuracy.Source: §5.2 · p. 8, Table 2 · p. 8
• RPTQ reaches 3-bit activation quantization, while the paper reports that SmoothQuant and PEG struggle at 4 bits.The authors attribute the difference to RPTQ accounting for activation range distributions across channels.Source: §5.2 · p. 8

5.3 Memory Consumption

• W4A8 reduces memory by about 63%, W4A4 by about 75%, W4A4KV by about 73%, and W3A3KV by about 80%.The reductions are reported for different settings; lower-bit activations are especially beneficial at high batch sizes and sequence lengths.Source: §5.3 · p. 9, Table 3 · p. 9

5.4 Ablation Study

• As cluster counts increase, perplexity generally decreases; the study uses 32 clusters for R1, R4, and R5, and 4 for R2 and R3.R2 fluctuations may be alleviated with more calibration data, while larger networks are more sensitive to cluster counts.Source: §5.4 · p. 9, Figure 4 · p. 9

• RPTQ addresses activation quantization by rearranging channels and quantizing them in clusters.The conclusion frames differing channel value ranges, rather than outliers alone, as the main challenge.Source: §6 · pp. 9–10

• OPT-30b and OPT-66b activations show range differences across layers and among channels within the same layer.The authors use this distribution analysis to motivate clustering channels with similar magnitude ranges to reduce quantization error.Source: §A.1 · p. 13

• SmoothQuant exhibits significant performance degradation on larger OPT networks, while PEG also incurs significant quantization loss on large models.The authors attribute SmoothQuant's degradation to larger channel-range differences in large networks; PEG was tested on OPT using the same group settings as RPTQ.Source: §A.2 · p. 13

A.3 Combine RPTQ with GPTQ

• GPTQ seeks to minimize weight-quantization error in transformer learner layers.Its optimization uses layer weights and inputs collected from a calibration dataset.Source: §A.3 · p. 13
• A Hessian-based approximation solves GPTQ’s layer-wise quantization problem using calibration inputs.Source: §A.3 · p. 13
• RPTQ reorders linear-layer weights, then applies GPTQ to each input cluster to minimize quantization error.Source: §A.3 · p. 13

A.4 Computation under cluster-based quantization

• Activation and weight clusters use separate asymmetric quantization parameters, allowing different clusters to be quantized independently.The activation clusters are formed after channel adjustment; weights are reordered according to the activation reorder index and divided into clusters.Source: §A.4 · p. 14
• The output can be computed either by dequantizing activations and weights before multiplication or directly in the integer domain, then dequantizing and summing results.The dequantized-value approach concatenates the cluster values before matrix multiplication.Source: §A.4 · p. 14
• The integer-domain method requires hardware support; most GPUs do not support 3-bit values, which are cast to supported 4-bit or 8-bit types.Source: §A.4 · p. 14

A.5 More Results for Ablation Study

• With 128/256 calibration samples, PT and C4 fluctuate in R2 and R3; the authors hypothesize limited calibration data may not represent some sample distributions.Larger calibration datasets would require more memory and computation, so the authors did not conduct that experiment.Source: §A.5 · p. 14

A.6 Memory Consumption of Different Parts

• The appendix reports memory proportions for different LLM components in Table 6.Source: §A.6 · p. 14, Table 6 · p. 15
• Figures 5 and 6 plot decoder-layer activation channels by maximum and minimum values, while Figure 7 tests cluster counts 1, 2, 4, 8, and 32 for R1–R5 under W16A4.The supplied passages describe these visuals but do not state their outcomes.Source: Figure 5 · p. 16, Figure 6 · p. 17, Figure 7 · p. 18