PaperlayerView exact revision on arXiv

Source-linked AI summary

BinaryBERT: Pushing the Limit of BERT Quantization

Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, Irwin King

arXiv:2012.15701v2cs.CL

TL;DR

Large pre-trained language models are difficult to deploy on edge devices because their size and computation burden increase, motivating more aggressive quantization. BinaryBERT uses ternary weight splitting to initialize and fine-tune a binary BERT, achieving slight performance loss while being 24× smaller on GLUE and SQuAD.

  • Problem

    Large pre-trained language models impose increasing model-size and computation costs that limit deployment on edge devices, while prior quantization reaches ternary but not binary weights.

  • Method

    BinaryBERT splits a trained ternary BERT’s quantized and latent weights equivalently to initialize a binary model, then fine-tunes it; adaptive splitting tailors model size to device constraints.

  • Results

    BinaryBERT achieves state-of-the-art BERT compression, with only a slight performance drop versus full-precision BERT-base while being 24× smaller on GLUE and SQuAD.

  • Takeaways & Limitations

    Ternary weight splitting makes binary BERT competitive across GLUE and SQuAD while supporting model sizes tailored to edge-device efficiency constraints.

  • Takeaways & Limitations

    DynaBERT squeezes width in Transformer layers but not the word-embedding layer, making the split binary model slightly larger than BWN.

Abstract

from arXiv · show

The rapid development of large pre-trained language models has greatly increased the demand for model compression techniques, among which quantization is a popular solution. In this paper, we propose BinaryBERT, which pushes BERT quantization to the limit by weight binarization. We find that a binary BERT is hard to be trained directly than a ternary counterpart due to its complex and irregular loss landscape. Therefore, we propose ternary weight splitting, which initializes BinaryBERT by equivalently splitting from a half-sized ternary network. The binary model thus inherits the good performance of the ternary one, and can be further enhanced by fine-tuning the new architecture after splitting. Empirical results show that our BinaryBERT has only a slight performance drop compared with the full-precision model while being 24x smaller, achieving the state-of-the-art compression results on the GLUE and SQuAD benchmarks.

1 Introduction

Quantization compresses BERT without redesigning its architecture, but reducing weights to 1-bit causes a sharp performance decline. BinaryBERT addresses this challenge with ternary weight splitting and achieves strong compression results across GLUE and SQuAD.

  • Motivation: Quantization replaces 32-bit parameters with low-bit representations without requiring a smaller model architecture.It can reduce model size and replace floating-point operations with lower-cost integer operations.
  • Challenge: 1-bit weight quantization causes a sharp performance drop, unlike the milder decline observed through 2-bit weights.The paper attributes this challenge to the binary model’s steep and complex loss landscape.
  • Approach: Ternary weight splitting initializes BinaryBERT by equivalently converting a trained ternary model’s quantized and latent weights.The initialized binary model retains ternary-model performance and can be refined through fine-tuning.
  • Approach: Adaptive splitting selects important ternary modules according to efficiency constraints, enabling binary models with flexible sizes for different edge devices.The method can leave other modules binary while splitting selected modules.
  • Results: 24× smaller models achieve only a slight performance drop versus full-precision BERT-base on GLUE and SQuAD.BinaryBERT split from a half-width ternary network also outperforms directly trained binary models with the original width.

2 Difficulty in Training Binary BERT

Directly training binary BERT is difficult because binarization creates a steep, complex, and irregular loss landscape. The paper analyzes this difficulty through performance comparisons, landscape visualizations, and Hessian-eigenvalue measurements.

  • Quantization background: Quantization-aware training quantizes latent full-precision weights during forward propagation and updates them with a straight-through estimator during backpropagation.The quantized weights determine the loss, while gradients update the latent weights despite the quantization function’s non-differentiability.
  • Quantization background: Weight binarization is non-trivial for BERT despite the appealing compression properties of binary networks.The paper contrasts binary weights with ternary values {±α, 0} and conventional binary scaling approaches.
  • Sharp performance drop: 3.8% and 0.9% performance drops occur on MRPC and MNLI-m, respectively, when reducing weight precision to 1-bit, compared with roughly 0.6% and 0.2% drops at 2-bit.The experiments use 8-bit activations and compare full-precision, 8-, 4-, 3-, 2-, and 1-bit weight quantization.
  • Loss-landscape analysis: Binary BERT has a higher and more complex loss surface than full-precision and ternary BERT, while the ternary surface is locally convex and easier to optimize.The binary surface is also steeper and more sensitive to perturbations, whereas the full-precision surface is flatter and more robust.
  • Quantitative analysis: λmax measures loss-surface steepness by approximating quantization-induced loss increase through the largest Hessian eigenvalue.The analysis evaluates Transformer parts separately, including MHA-QK, MHA-V, MHA-O, FFN-Mid, and FFN-Out.
  • Quantitative analysis: ∼15× larger top-1 eigenvalues occur for binary MHA-O than for its full-precision counterpart, with binary eigenvalues higher in expectation and standard deviation overall.These larger eigenvalues imply less tightly bounded quantization-loss increases and greater optimization difficulty for binary models.

3 Proposed Method

Ternary weight splitting initializes BinaryBERT by equivalently converting a trained ternary model’s quantized and latent weights into binary counterparts, then fine-tunes the expanded architecture. Adaptive splitting allocates ternary precision according to sensitivity and resource constraints before producing an all-binary model.

  • 3.1 Ternary Weight Splitting: Ternary weight splitting uses a trained ternary model as an optimization proxy for BinaryBERT’s challenging binary loss landscape.The method exploits the flatter ternary landscape to initialize the binary model.
  • 3.1 Ternary Weight Splitting: BinaryBERT is obtained by fine-tuning the full-width binary architecture after splitting from a half-sized ternary model.Fine-tuning addresses the possibility that the ternary model’s minimum does not remain optimal after architectural expansion.
  • 3.1 Ternary Weight Splitting: The method splits both quantized and latent ternary weights into two binary matrices while preserving equivalent outputs for identical inputs.The latent weights additionally satisfy wt = wb1 + wb2 after splitting.
  • 3.1 Ternary Weight Splitting: Quantization uses layer-wise ternarization for Transformer matrices, row-wise ternarization for word embeddings, and separate scaling factors after splitting.Activations are quantized before matrix multiplications, while layer normalization, skip connections, bias, and the final classification layer are skipped or left unquantized.
  • 3.2 Adaptive Splitting: Adaptive splitting adjusts BinaryBERT’s width using parameter sensitivity and edge-device resource constraints.Sensitive parts can remain ternary during intermediate training before being split into binary weights.
  • 3.2 Adaptive Splitting: The splitting assignment represents ternary choices with a binary vector and optimizes sensitivity under parameter or FLOPs costs, with dynamic programming avoiding NP-hardness.The sensitivity vector measures performance gains from leaving matrices unquantized, while the cost vector records additional parameter or FLOPs increases.

4 Experiments

BinaryBERT is evaluated on GLUE and SQuAD under low-bit weights and activations, with comparisons against binary training, adaptive splitting strategies, and other compression methods. Results show consistent gains from ternary weight splitting, further improvement after fine-tuning, and competitive performance at substantially reduced model size.

  • Experimental setup: GLUE and SQuAD experiments evaluate task performance alongside model size and inference computation under quantized weights and activations.GLUE reports task-specific metrics and SQuAD reports EM and F1; model size and bit-wise FLOPs are also measured.
  • GLUE results: Ternary weight splitting consistently outperforms BWN on GLUE development and test results with both 8-bit and 4-bit activation quantization.The advantage is larger for 4-bit activations than for 8-bit activations.
  • SQuAD results: 1.8% and 0.6% EM improvements from 4-bit activation splitting occur on SQuAD v1.1 and v2.0, respectively, exceeding the corresponding 8-bit gains.The proposed method also outperforms BWN on both EM and F1 across both datasets.
  • Adaptive splitting: Maximal Gain adaptive splitting compares favorably with Random Gain and Minimal Gain across model sizes using average performance over six GLUE tasks.The evaluated model-size endpoints are 9.8MB and 16.5MB for half-sized and full-sized BinaryBERT.
  • State-of-the-art comparison: 24× compression compared with BERT-base is achieved with only 0.4% MNLI-m and 0.0%/0.2% SQuAD v1.1 performance drops.These comparisons are reported for BinaryBERT against the full-precision model.
  • Ablation and analysis: Fine-tuning after splitting consistently improves performance for both 8-bit and 4-bit activation settings.The experiment converts a half-width ternary model into a full-sized binary model before further refinement.
  • Ablation and analysis: TWS attains lower MRPC training loss than BWN and surpasses TWN after resetting optimizers and learning-rate schedulers for fair comparison.Optimization trajectories indicate binary models move toward the optimal solution for both 8-bit and 4-bit activations.

5 Related Work

Related work covers network binarization, mixed-precision quantization, and neuron splitting. BinaryBERT combines the deployment simplicity of binary models with splitting-based initialization and adaptive sizing.

  • Network binarization: Network binarization has primarily been explored in computer vision through quantization-error reduction, training-objective improvements, and gradient-mismatch reduction.The paper notes limited exploration of binarization for pre-trained language models.
  • Mixed-precision quantization: Mixed-precision quantization reallocates layer-wise bit-width according to sensitivity to improve compression, but its resulting models are harder to deploy.Existing approaches use differentiable search, reinforcement learning, or loss curvature.
  • Mixed-precision quantization: Binary architecture-search methods pursue similar goals but are usually too time-consuming for large pre-trained language models.This limits their practicality for BERT-scale compression.
  • Neuron splitting: Neuron splitting progressively increases network width while inheriting knowledge from earlier networks, and has also been applied to quantization.Quantization-oriented splitting can remove full-precision outliers by splitting neurons with large magnitudes.

6 Conclusion

The paper concludes that BinaryBERT addresses the optimization difficulty of BERT weight binarization through ternary weight splitting and subsequent fine-tuning. Adaptive splitting further adjusts model size for edge-device constraints while achieving strong compression results.

  • 6 Conclusion: Directly training BinaryBERT is difficult because binarization produces a steep and complex loss landscape with a large performance drop.Ternary weight splitting instead initializes a binary model from a trained ternary BERT.
  • 6 Conclusion: Ternary weight splitting followed by fine-tuning improves over vanilla binary training and supports adaptive model sizing.The adaptive variant tailors BinaryBERT to edge-device constraints.
  • 6 Conclusion: BinaryBERT achieves state-of-the-art BERT compression performance according to the paper’s empirical results.

A Derivation of Equation (8)

The appendix derives the parameters used in Equation (8) from the BWN quantizer under assumptions on a and b. The derivation establishes solutions satisfying 0 < a < 1 and b > 0.

  • Derivation: The derivation obtains a and b by recalling the BWN quantizer and simplifying it under 0 < a < 1 and b > 0.
  • Derivation: The resulting expression gives the solution for a, which is empirically observed to satisfy 0 < a < 1.
  • Derivation: The subsequent derivation obtains b from the weight relation and verifies that the solution satisfies b > 0.

B Implementation Details

Adaptive splitting estimates parameter-wise quantization sensitivity and uses dynamic programming to allocate splitting under a storage constraint. Experiments examine binarization across precisions, distillation stages, and Transformer components.

  • Adaptive splitting: Quantization sensitivity is measured by parameter-wise performance gain for Transformer parts and layers, with embedding and pooler sensitivities computed similarly.The gain from leaving a component unquantized is normalized by its parameter count.
  • Adaptive splitting: Dynamic programming solves the adaptive-splitting combinatorial optimization by treating the storage constraint as knapsack capacity and sensitivity scores as item values.The baseline efficiency is that of the half-sized binary network.
  • Evaluation: Multi-bit experiments evaluate GLUE performance with both 8-bit and 4-bit activation quantization across repeated random seeds.The results are presented in Figures 8 and 9 and compare performance across weight bit-widths.
  • Evaluation: Additional visualizations compare loss landscapes for MHA-K, MHA-O, FFN-Mid, and FFN-Out in the first two Transformer layers.MHA-Q is omitted because it is described as symmetric to MHA-K.

C.3 Ablation of Knowledge Distillation

The ablation compares knowledge-distillation settings after ternary weight splitting. Intermediate- plus prediction-layer distillation performs better than either no distillation or prediction-only distillation.

  • Distillation settings: The ablation compares no distillation, prediction-layer distillation, and the default combination of intermediate- and prediction-layer distillation.Fine-tuning after splitting uses corresponding distillation settings for the compared configurations.
  • Results: “Int.+Pred.” outperforms both “N/A” and “Pred.” with a clear margin after splitting.The experiment omits data augmentation.

C.4 Detailed Results of Adaptive Splitting

Adaptive splitting generally performs best by assigning ternary precision to the most quantization-sensitive modules. This pattern holds across model sizes and both activation-precision settings.

  • Adaptive splitting: The strategy that splits the most sensitive modules mostly performs best on average across various model sizes.It is compared with random and minimal-gain strategies under both 8-bit and 4-bit activation quantization.

C.5 Architecture Visualization

Architecture visualizations show how adaptive splitting distributes split parameters across Transformer layers as model size increases. Shallower layers receive preference over deeper layers.

  • Architecture visualization: With increasing model size, adaptive splitting prefers shallower Transformer layers over deeper layers.This agrees with the performance-gain ordering reported in Figure 7.
  • Architecture visualization: The 9.8MB configuration represents no splitting, whereas 16.5MB represents splitting all splittable model parameters.These are the half-sized and full-sized BinaryBERT endpoints.
  • Loss-landscape context: Figures 10–13 visualize loss landscapes for MHA-K, MHA-Out, FFN-Mid, and FFN-Out in the first two Transformer layers on MRPC.Each figure corresponds to one Transformer parameter group.
  • Architecture visualization: Figure 14 records the number of parameters split in each layer rather than storage.The visualization merges all splittable parameters within each Transformer layer.
Loading 2012.15701v2…