Training Transformers with 4-bit Integers

Motivation

• Fully quantized training seeks lower computational and memory costs, but existing 4-bit methods use custom numerical formats unsupported by contemporary hardware.Low-precision training also faces rugged optimization and inaccurate gradients that can slow or destabilize convergence.Source: §1 · p. 1

Method

• The method targets transformer training with all costly linear operations represented as matrix multiplications and quantized using structures specific to activations, weights, and gradients.Its quantizers draw on randomized numerical linear algebra.Source: §1 · pp. 1–2

Forward propagation

• Hadamard quantization transforms activations to spread outlier information across nearby entries and reduce their numerical range during forward propagation.Activation outliers are identified as the main source of forward-quantization accuracy degradation.Source: §1 · p. 2

Backpropagation

• Bit splitting and leverage score sampling exploit gradient sparsity by prioritizing informative large-token gradients over very small ones during backpropagation.Each token gradient is divided into higher and lower 4-bit components before importance sampling.Source: §1 · p. 2

Results

• Up to 2.2 times faster linear operators and 35.1% faster training are achieved while supporting INT4 matrix multiplications on contemporary GPUs.The method attains competitive or superior accuracy across natural language understanding, question answering, machine translation, and image classification.Source: §1 · p. 2

Fully Quantized Training

• Existing 4-bit fully quantized training methods still show approximately 1–2.5% accuracy drops on several tasks and cannot support contemporary hardware.The frontier includes quantizing activations, weights, and gradients with specialized formats and algorithms.Source: §2 · p. 2

Other Efficient Training Methods

• Mixture-of-experts, structural dropout, efficient attention, and distributed training address capacity, regularization, attention complexity, or resource scaling rather than numerical precision.The paper characterizes its precision-reduction approach as orthogonal to these directions.Source: §2 · p. 2

3 Forward Propagation

• INT4 arithmetic accelerates all transformer linear operators, while less-computationally-intensive nonlinear operators remain in FP16.The discussion formulates linear layers as matrix multiplications and notes applicability to batch matrix multiplications.Source: §3 · pp. 2–3

3.1 Learned Step Size Quantization

• Learned step size quantization uses a static, learnable scale to map FP matrices to integer values and approximate XW⊤ with scaled INT4 matrix multiplication.For INT4, the quantizer uses QN = QP = 7 and dequantizes the INT32 product back to FP16.Source: §3.1 · p. 3

3.2 Activation Outliers

• 4-bit forward quantization is difficult because transformer activations contain large outliers, forcing a trade-off between representing outliers and preserving granularity for typical entries.This issue is especially significant when fine-tuning pretrained models, whose activations contain more outliers.Source: §3.2 · p. 3

3.3 Hadamard Quantization

• Hadamard quantization maps matrices into a linear space with fewer outliers by spreading concentrated feature-wise outliers across entries.The block-diagonal transform uses Hk blocks and preserves the product through inverse-transform cancellation.Source: §3.3 · p. 4
• HQ-MM adds FP16 transformation and dequantization work but remains cheaper than FP16 matrix multiplication when 2^k is much smaller than both matrix dimensions.The typical block parameter is k = 5, with dimensions commonly ranging from 768 to 4096.Source: §3.3 · p. 4

4 Backpropagation

• The backpropagation pipeline applies these quantizers to the required gradient products for both weights and activations, alongside element-wise masks and Hadamard-related multiplications.The third matrix-multiplication type quantizes the FP16 output gradient, while previously quantized activations and weights are reused from forward propagation.Source: §4 · p. 5, §4.2 · p. 7

4.1 Structural Sparsity of Gradients

• Structural gradient sparsity emerges because most training examples are well fitted, leaving only a few token rows with large activation gradients.The sparsity appears quickly during pretraining and persists throughout fine-tuning.Source: §4.1 · p. 5

4.2 Bit Splitting and Leverage Score Sampling

• The method drops small gradient rows and uses the saved computation to represent large rows more accurately, addressing the limited fidelity of direct INT4 quantization.This trades low-impact components for improved representation of large gradient rows.Source: §4.2 · pp. 5–6
• Bit splitting represents each gradient matrix with higher and lower 4-bit components, enabling its products to use two INT4 matrix multiplications.The resulting representation can be viewed as an INT8 matrix split into two INT4 matrices.Source: §4.2 · p. 5
• Leverage score sampling assigns probabilities to rank-1 gradient components and samples them with Bernoulli masks, producing an unbiased approximation with half as many nonzero components in expectation.The sampling probabilities are chosen in proportion to leverage scores, which minimizes the approximation variance.Source: §4.2 · pp. 6–7
• The resulting gradient products retain INT4 matrix-multiplication cost comparable to naïve quantization, while the other sampling and reconstruction steps add O(NC + ND) overhead.For the weight-gradient procedure, the two sampled products take about 2NCD INT4 multiply-accumulates.Source: §4.2 · pp. 6–7

5.1 Converged Model Accuracy

• 5.5% and > 25% average accuracy improvements over LSQ+LUQ are achieved on BERT-base and BERT-large GLUE fine-tuning, respectively.HQ+LSS also improves SQuAD, SQuAD 2.0, Adversarial QA, SWAG, and CoNLL-2003 relative to LSQ+LUQ.Source: §5.1 · p. 8
• HQ+LSS has about 1.0% BLEU degradation, compared with 2.1% for Ultra-low and 0.3% reported for LUQ, while supporting contemporary hardware.The comparison uses differing BLEU conventions across the reported methods.Source: §5.1 · p. 8
• HQ+LSS improves image-classification accuracy over LSQ+LUQ on CIFAR-100 and ImageNet1k while keeping CIFAR-10 degradation below 0.5%.For ImageNet1k, reported improvements are 2%, 2.6%, and 0.2% across the listed ViT configurations; DeiT-Small pretraining reaches similar accuracy to LSQ+LUQ.Source: §5.1 · p. 8

5.2 Ablation Study

• Maintaining the largest 1% of activation entries performs well, confirming that activation outliers are the main forward-quantization challenge; HQ matches this upper-bound approach and outperforms LSQ.More granular quantizers and SmoothQuant do not achieve good 4-bit accuracy without HQ.Source: §5.2 · pp. 8–9
• LSS matches or slightly exceeds LUQ when backward bit-width exceeds 2, while requiring only INT4 arithmetic.The ablation compares minimax quantization, LUQ, and LSS with forward propagation retained in FP16.Source: §5.2 · p. 9

5.3 Computational and Memory Efficiency

• INT4 operators run up to 2.2 times faster than FP16 matrix multiplication, and INT4 training reaches up to 35.1% speedup for BERT-style and 26.5% for GPT-style models.The operator comparison uses an RTX 3090, while training throughput is compared on eight A100 GPUs; the implementation is explicitly described as prototypical and not fully optimized.Source: §5.3 · p. 9

6 Conclusions

• HQ and LSS quantize activations and gradients for hardware-friendly INT4 training, with comparable or better performance than existing INT4 methods on several tasks.The approach may extend to other matrix-multiplication-only architectures, including MLP-Mixer, graph neural networks, and recurrent neural networks.Source: §6 · p. 9
• The method’s main limitation is that it accelerates models with many matrix multiplications but cannot accelerate convolution layers, and it does not yet work well for extremely large models such as OPT-175B.The paper notes that even INT8 training remains an open problem for such large models.Source: §6 · p. 9

A.1 BMM in Attention

• BMMs in attention can use the same HQ and LSS methods as ordinary matrix multiplications.The construction defines batched tensors and leverage scores so both quantizers apply to BMMs.Source: §A.1 · p. 14

A.2 Computing Leverage Score

• The leverage-score probability algorithm repeatedly clamps probabilities above one and rescales smaller values until a valid distribution is obtained in at most O(N) iterations.The number of probabilities exceeding one decreases by at least one after each unsuccessful scaling step.Source: §A.2 · pp. 14–15

A.3 Learning Quantizer Parameters

• Quantizer step sizes are learned through chain-rule gradients using previously computed matrix products and inexpensive elementwise residual calculations.The residual terms are formed from quantized tensors and their quantization errors, while the most expensive matrix multiplications dominate step-size computation.Source: §A.3 · p. 15
• LSS-MM quantizes gradients with bit splitting, computes leverage scores, samples rows, and combines dequantized INT32 products into an FP16 result.The sampled matrix multiplications require about 2NCD INT4 MACs in expectation.Source: §A.3 · p. 15

A.4 Cold Start Problem

• During the cold start, step sizes are not learned for the first few iterations because rapidly changing weights and activations can make joint optimization unstable.A heuristic dynamically sets each tensor’s step size during this initial period.Source: §A.4 · p. 15

A.5 Choose hadamard matrix size

• Hadamard matrix size is selected by minimizing the product of activation and weight quantization errors, then fixed for fine-tuning or reinitialized after roughly two pretraining epochs.The transformed tensors are treated as approximations of the original activation and weight matrices.Source: §A.5 · p. 15

A.6 GPU Implementation

• HQ-MM consists of Hadamard multiplication, quantization, packing, INT4 GEMM, and dequantization, while LSS-MM adds leverage-score calculation, sampling, and LSQ backpropagation.These components expose the principal computational stages implemented on the GPU.Source: §A.6 · p. 16
• Two INT4 values are packed into one INT8 value before Cutlass Tensor Core GEMM, then the result is dequantized back to FP16.The packing operation combines adjacent 4-bit values into a single 8-bit storage unit.Source: §A.6 · p. 16
• INT4 Tensor Core GEMM requires row-major and column-major inputs, so transpose-plus-contiguous operations are needed and remain time-consuming.The implementation identifies this layout conversion as requiring further optimization.Source: §A.6 · p. 16
• The implementation fuses higher- and lower-4-bit quantization and uses quantized INT8 data for leverage-score calculation before scaling the final score.Rows or columns with nonzero Euclidean norm are sampled in practice because this can outperform the repeated probability procedure.Source: §A.6 · p. 16
• After leverage-score sampling, matrices may require zero-padding so the GEMM dimension K becomes a multiple of 32.This alignment requirement arises in backpropagation because sampling can change matrix shapes.Source: §A.6 · p. 16

B. Proofs.

• The leverage-score proof establishes the variance formulation for the weight-gradient estimator.The supplied proof section identifies this result as Proposition B.1.Source: B Proofs · p. 17, §B.1 · p. 17

B.2 Proof of Activation Leverage Score in Sec. 4.2

• LSS assigns each gradient row a probability in [0, 1] and forms an unbiased estimator from sampled rows.The construction divides the matrix multiplication into smaller products and uses leverage scores to determine sampling probabilities.Source: §B.2 · pp. 17–18
• Discarded rows become zero in the sampled product, so their multiplications need not be computed.In the described case, this removes half of the computation.Source: §B.2 · p. 18

C Experiments.

• The experiments section introduces additional implementation details for the paper’s evaluations.The supplied passage marks this material as the beginning of the experiments appendix.Source: C Experiments · p. 18

C.1 Experiments setup

• The experiments implement the algorithm with Transformers for GLUE, QA, SWAG, and CoNLL, Fairseq for machine translation, and separate repositories for ViT and DeiT.These codebases support the paper’s language, translation, fine-tuning, and pretraining evaluations.Source: §C.1 · p. 18
• Most experiments use NVIDIA GeForce RTX 3090, while BERT-Large and ViT-L use NVIDIA A40.Runtime measurements use NVIDIA T4, 3090, and A100 GPUs.Source: §C.1 · p. 18
• Table 2 compares FP, INT8, LSQ + LUQ, and HQ + LSS training on GLUE for BERT-base-uncased and BERT-large uncased.The caption defines HQ + LSS as Hadamard quantization for forward propagation and leverage score sampling for backward propagation.Source: Table 2 · p. 19

• For BERT-base, HQ+LSS has below 0.5% accuracy degradation on STSB, SST2, QNLI, and QQP, with a smaller gap on challenging CoLA and RTE.Source: §C.2 · p. 19
• Accuracy improvement over LSQ+LUQ exceeds 30% on CoLA, QNLI, and MNLI for BERT-large.Source: §C.2 · p. 19
• On BERT-large, accuracy improvements exceed 10% on SST2 and QQP, exceed 5% on RTE, and exceed 3% on STSB and MRPC.Source: §C.2 · p. 19
• The advantage over LSQ+LUQ becomes more significant as task difficulty increases, potentially because challenging tasks store more information in activation outliers.Source: §C.2 · p. 19

C.3 More Granular Quantization Methods

• Per-token, per-channel, and SmoothQuant variants do not work under 4-bit FQT, and combining them with HQ brings no significant improvement.Source: §C.3 · p. 19
• LSQ benefits all tested granular quantization methods, whereas SmoothQuant can harm LSQ at low bit-width because it sacrifices inlier precision to preserve outliers.Source: §C.3 · pp. 19–20

C.4 Large Language Model Operator Speed

• Table 3 reports total time for epochs 1–5 on GPT2-base and BERT-large experiments.Source: Table 3 · p. 20, §C.4 · p. 20

• Figure 6 decomposes HQ-MM and LSS-MM runtime into their component operations as input-matrix shapes vary.Source: §C.5 · p. 20
• Figures 7 and 8 report real quantization performance on Nvidia T4 and A100 GPUs, respectively.Source: Figure 7 · p. 21, Figure 8 · p. 21, §C.5 · p. 22
• Table 4 compares activation-and-weight quantization methods from 2 to 8 bits, including per-token activation and per-channel weight quantization.Source: Table 4 · p. 21
• LSS-MM includes higher- and lower-bit quantization, leverage-score computation, sampling, dequantization, and LSQ backpropagation; LSS-Act omits quantization and leverage computation for timing.Source: §C.5 · p. 21
• The hardware implementation is evaluated on RTX 3090, T4, and A100 GPUs with different CUDA capabilities.Source: §C.5 · p. 22