PaperlayerView exact revision on arXiv

Source-linked AI summary

BiBERT: Accurate Fully Binarized BERT

Haotong Qin, Yifu Ding, Mingyuan Zhang, Qinghua Yan, Aishan Liu, Qingqing Dang, Ziwei Liu, Xianglong Liu

arXiv:2203.06390v1cs.CL

TL;DR

Full binarization could make BERT more compact, but accurately binarizing weights, embeddings, and especially activations remained a challenge. BiBERT combines Bi-Attention and Direction-Matching Distillation to address information loss and optimization mismatch. On GLUE, it outperforms existing ultra-low-bit-activation quantized BERTs while reporting 56.3× FLOPs and 31.2× model-size savings.

  • Problem

    Prior work had not accurately binarized BERT’s weights, embeddings, and activations together, although full binarization offers compact parameters and bitwise operations.

    Source: §1 · p. 1

  • Method

    BiBERT pairs information-entropy-oriented Bi-Attention with Direction-Matching Distillation using appropriate activations and similarity-matrix knowledge.

    Source: §1 · p. 2

  • Results

    BiBERT outperforms existing quantized BERTs with ultra-low-bit activations on GLUE, with 56.3× FLOPs and 31.2× model-size savings.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The reported savings and GLUE results indicate potential for fully binarized BERT in resource-constrained deployment scenarios.

    Source: §1 · p. 2

    Binarized attention weights take negative values unlike the original normalized [0, 1] attention weights, a design the authors report is harmful to the attention structure.

    Source: §3.2.1 · p. 5

Abstract

from arXiv · show

The large pre-trained BERT has achieved remarkable performance on Natural Language Processing (NLP) tasks but is also computation and memory expensive. As one of the powerful compression approaches, binarization extremely reduces the computation and memory consumption by utilizing 1-bit parameters and bitwise operations. Unfortunately, the full binarization of BERT (i.e., 1-bit weight, embedding, and activation) usually suffer a significant performance drop, and there is rare study addressing this problem. In this paper, with the theoretical justification and empirical analysis, we identify that the severe performance drop can be mainly attributed to the information degradation and optimization direction mismatch respectively in the forward and backward propagation, and propose BiBERT, an accurate fully binarized BERT, to eliminate the performance bottlenecks. Specifically, BiBERT introduces an efficient Bi-Attention structure for maximizing representation information statistically and a Direction-Matching Distillation (DMD) scheme to optimize the full binarized BERT accurately. Extensive experiments show that BiBERT outperforms both the straightforward baseline and existing state-of-the-art quantized BERTs with ultra-low bit activations by convincing margins on the NLP benchmark. As the first fully binarized BERT, our method yields impressive 56.3 times and 31.2 times saving on FLOPs and model size, demonstrating the vast advantages and potential of the fully binarized BERT model in real-world resource-constrained scenarios.

1 INTRODUCTION

BERT’s resource demands motivate compression, but full binarization remains difficult, especially for activations. BiBERT targets the resulting information and optimization bottlenecks and reports strong GLUE performance with substantial compute and model-size savings.

1 INTRODUCTION

  • Full binarization uses 1-bit weights, embeddings, and activations, but 1-bit activation binarization causes the most severe drop and prior work had not achieved it accurately.BERT’s large parameter count also hinders deployment on edge devices.Source: §1 · pp. 1–2
  • The baseline’s performance loss is attributed to attention-information degradation in forward propagation and distillation optimization-direction mismatch in backward propagation.Directly binarized attention weights lose information; distillation using attention scores produces a non-negligible error between actual and expected optimization directions.Source: §1 · p. 2
  • BiBERT combines information-entropy-oriented Bi-Attention with Direction-Matching Distillation, which uses appropriate activations and similarity-matrix knowledge to improve optimization.Bi-Attention uses binarized representations with maximized information entropy to restore perception of input content.Source: §1 · p. 2
  • 56.3× FLOPs and 31.2× model-size savings accompany BiBERT’s GLUE results, which outperform existing quantized BERTs with ultra-low-bit activations.The paper also reports average accuracy exceeding 1-1-1-bit BinaryBERT by 20.4% and 2-8-8-bit Q2BERT by 13.3%.Source: Figure 1 · p. 1, §1 · p. 2

2 BUILDING A FULLY BINARIZED BERT BASELINE

The paper establishes a fully binarized BERT baseline by binarizing representations in the forward pass and applying teacher-student distillation during optimization. It describes the model’s transformer computation and layerwise distillation targets.

2 BUILDING A FULLY BINARIZED BERT BASELINE

  • The baseline binarizes BERT representations during forward propagation and applies distillation during backward optimization.It is presented as a straightforward solution using common techniques.Source: §2 · p. 2

2.1 BINARIZED BERT ARCHITECTURE

  • Binarized BERT embeds inputs before transformer blocks, where each attention head derives queries, keys, and values from hidden states and computes attention scores.The transformer blocks retain multi-head attention and feed-forward components; the rest follows the original BERT architecture.Source: §2.1 · p. 3

2.2 DISTILLATION FOR BINARIZED BERT

  • Layerwise distillation matches student attention scores, MHA outputs, and hidden states to full-precision teacher features using mean squared error.Prediction-layer distillation additionally minimizes soft cross-entropy between student and teacher logits.Source: §2.2 · p. 3

3 THE RISE OF BIBERT

BiBERT identifies information degradation in forward propagation and optimization-direction mismatch in backward propagation as the fully binarized BERT baseline’s bottlenecks. It addresses them with entropy-maximizing Bi-Attention and Direction-Matching Distillation.

3 THE RISE OF BIBERT

  • The baseline’s bottlenecks lie in architecture during forward propagation and optimization during backward propagation.BiBERT’s proposed remedies are Bi-Attention for information degradation and DMD for direction mismatch.Source: §3 · p. 4, §3.1 · p. 4

3.1 BOTTLENECKS OF FULLY BINARIZED BERT BASELINE

  • Binarizing MHA drops accuracy by up to 11.49% on SST-2 (full-precision 89.09% vs. binarized 77.60%), making attention the highest-priority structure to improve.Removing attention-score distillation instead raises accuracy by 0.81% on SST-2 and 1.10% on QQP, motivating a different distillation design.Source: §3.1 · p. 4

3.2 BI-ATTENTION

  • Bi-Attention statistically maximizes the entropy of binarized representations to mitigate attention information loss and revive the attention mechanism.The design also applies bitwise operations for fast inference.Source: §3.2 · p. 4, §3.2.2 · p. 6

3.2.1 INFORMATION DEGRADATION IN ATTENTION STRUCTURE

  • Directly sign-binarizing softmax attention weights maps every element to 1, reducing their entropy to zero.A fixed shift threshold is difficult to choose statistically when attention-mask lengths vary, and binarized weights can be negative despite the original [0, 1] range.Source: §3.2.1 · p. 5

3.2.2 BI-ATTENTION FOR MAXIMUM INFORMATION ENTROPY

  • Because entropy-maximized binarized query-key scores are symmetric and approximate N(0, D), Bi-Attention can binarize scores at threshold zero.It then uses Boolean attention weights to filter lower-valued elements and a bitwise-affine matrix multiplication operator for efficient calculation.Source: §3.2.2 · p. 6

3.3.1 DIRECTION MISMATCH

  • For standard-normal activations, 1-bit binarization produces 14.4% optimization-direction error, about 4.4× the 3.3% error for 4-bit quantization.Distilling attention scores formed from binarized queries and keys accumulates this mismatch; mismatch exceeds match on average, 0.64 vs. 0.45.Source: §3.3.1 · pp. 6–7

3.3 DIRECTION-MATCHING DISTILLATION

  • DMD addresses direction mismatch in distillation by selecting activations and constructing similarity matrices to improve optimization of fully binarized BERT.The scheme is designed to provide matching optimization directions and reduce scale differences and numerical instability.Source: §3.3.2 · p. 7

3.3.2 DIRECTION-MATCHING DISTILLATION FOR ACCURATE OPTIMIZATION

  • DMD distills upstream queries and keys instead of attention scores, also distilling values to cover all inputs to multi-head attention.It uses scale-normalized, numerically stable similarity patterns for student-teacher distillation.Source: §3.3.2 · p. 7

4 EXPERIMENTS

BiBERT improves fully binarized BERT through Bi-Attention and DMD, with experiments covering ablations, GLUE comparisons, and efficiency and training analyses. The reported comparisons show gains over other quantized methods, while analysis examines representation entropy and training convergence.

4.1 ABLATION STUDY

  • Bi-Attention and DMD each improve the fully binarized baseline on SST-2, MRPC, RTE, and QQP, and combining them boosts performance further.The authors say the two techniques promote each other and close the gap with the full-precision counterpart.Source: §4.1 · p. 8, Table 1 · p. 8

4.2 COMPARISON WITH SOTA METHODS

  • BiBERT outperforms other methods on GLUE development sets, and with data augmentation reaches 90.9% accuracy on SST-2.The reported 90.9% is on SST-2; the authors also report that BiBERT lags Q-BERT (2-8-8) and TernaryBERT (2-2-2) on MNLI and STS-B with data augmentation.Source: §4.2 · p. 8, Table 3 · p. 9
  • 56.3× FLOPs and 31.2× model size savings over full-precision BERT accompany BiBERT’s accuracy results.The authors attribute the computation and storage savings relative to other quantized BERTs to Bi-Attention’s bit-operation BAMM replacing softmax.Source: §4.2 · p. 9

4.3 MORE ANALYSIS

  • BiBERT’s attention-weight entropy stays near maximized during training, while the baseline’s degrades to 0.The reported pattern is observed in the first head of layer 0 and is said to occur across all heads and layers.Source: §4.3 · p. 9
  • On SST-2 without data augmentation, BiBERT converges faster and achieves higher accuracy than the fully binarized baseline.The comparison is based on the training-loss curves.Source: §4.3 · p. 9

5 CONCLUSION

The paper concludes that BiBERT addresses bottlenecks in fully binarized BERT with Bi-Attention and DMD. Its appendix explains how zero-mean weight redistribution can maximize information entropy in binarized weights and activations.

5 CONCLUSION

  • BiBERT combines Bi-Attention and DMD to improve fully binarized BERT and outperforms existing BERT quantization methods with ultra-low-bit activations.The conclusion reports 56.3× FLOPs and 31.2× model size savings.Source: §5 · p. 9

A.1 DISCUSSION OF BINARIZED LINEAR LAYER

  • For zero-mean Gaussian X, the entropy of sign-binarized BX is maximized; zero-mean weight redistribution is used to maximize entropy in weights and activations.The appendix says zero-mean weights make the next-layer output mean zero, so its activation entropy is maximized under Proposition 1.Source: §A.1 · p. 12

A.2 PROOF OF THE THEOREM 1

The proof establishes that the threshold maximizing binarized attention entropy decreases as the number of elements increases. It converts an after-softmax threshold into a before-softmax threshold to show this relationship.

A.2 PROOF OF THE THEOREM 1

  • The entropy-maximizing threshold τ for sign(softmax(A) − τ) is negatively correlated with the number of elements k.The proof compares thresholds for k and k+1 elements and derives τ_k+1 < τ_k.Source: §A.2 · pp. 13–14
  • The proof converts an after-softmax threshold τ_k into a before-softmax threshold for A_1, using the order-preserving property of softmax.It identifies a unique threshold based on the other elements A_2 through A_k.Source: §A.2 · p. 13

A.3 PROOF OF THE THEOREM 2

The appendix characterizes binarized attention scores as approximately normally distributed and describes a bitwise operation for computing binarized attention during inference.

A.3 PROOF OF THE THEOREM 2

  • With entropy-maximized binarized queries and keys, each attention score A_ij has a probability mass function approximating N(0, D).The proof treats query-key element products as Bernoulli trials and invokes a normal approximation to the resulting binomial distribution.Source: §A.3 · pp. 14–15
  • The proof states that the distribution of the output is approximately N(0, D).It derives this through calculations of the output distribution’s mean and standard variation.Source: §A.3 · p. 16

A.4 BAMM OPERATION IN BI-ATTENTION

  • The proposed ⊠ bitwise operation computes between binarized attention weights and binarized values during inference.The operation supports the hardware representation in which 1-bit matrices use values 1 and −1; it can be simulated by multiplication during training.Source: §A.4 · p. 16

A.5 PROOF AND DISCUSSION OF THE THEOREM 3

Theorem 4 shows that symmetric quantization increasingly mismatches optimization directions as bit-width decreases. The paper's analysis and simulation both indicate that the mismatch grows sharply at 1-bit quantization.

A.5 PROOF AND DISCUSSION OF THE THEOREM 3

  • As Q decreases from 8 to 1, the proportion of optimization direction error increases under symmetric quantization.The theorem defines the error as a sign disagreement between the original and quantized optimization directions.Source: §A.5 · p. 17
  • The analysis finds that direction error increases as Q becomes smaller within the quantization range, while the direction remains correct beyond its bounds.For X > L or X < −L, the proof states the optimization direction is always right.Source: §A.5 · p. 18
  • Monte Carlo simulation experimentally confirms the theorem and shows direction mismatch probability rises rapidly with 1-bit quantization.The simulation samples 10,000 standard-normal values with quantization range [−1, 1].Source: §A.5 · p. 18, Table 4 · p. 18

A.6 PROOF AND DISCUSSION OF THE EQUIVALENCE

The appendix establishes that a threshold on attention scores can preserve the entropy-maximizing binarization obtained from softmax attention weights. It also specifies GLUE evaluation metrics and reports inference FLOPs and model size.

A.6 PROOF AND DISCUSSION OF THE EQUIVALENCE

  • An appropriate threshold φ(τ, A) makes entropy maximization of sign(A − φ(τ, A)) equivalent to maximizing entropy of sign(softmax(A) − τ).The proof uses the order-preserving property of softmax; for A distributed as N(0, D), a fixed threshold of zero maximizes binarized attention-weight entropy.Source: §A.6 · pp. 18–19

B.1 DATASET AND METRICS

  • GLUE evaluation uses task-standard metrics, excludes WNLI, and reports inference FLOPs and model size alongside task performance.STS-B uses Spearman correlation, CoLA uses Matthews correlation coefficient, and the other tasks use accuracy; MNLI-m and MNLI-mm are both reported.Source: §B.1 · p. 19

B.2 IMPLEMENTATION

The implementation distills a directly binarized DynaBERT from a full-precision teacher and uses task-specific training settings. The baseline choices prioritize low inference overhead and representative binarization techniques.

B.2 IMPLEMENTATION

  • The baseline uses full-precision DynaBERT as teacher, directly binarizes its parameters, and applies sign in the forward pass, STE in backpropagation, and MSE distillation.The BERT-base setting has 12 transformer layers, hidden size 768, and 12 attention heads; training epochs vary by GLUE task.Source: §B.2 · pp. 19–20
  • Under the same 1-1-1 setting, BiBERT's sign-binarized activation outperforms methods using real-time scaling factors while requiring less computation.The comparison is with TernaryBERT and BinaryBERT, which apply real-time calculated activation scaling factors.Source: §B.2 · p. 20

C.1.1 REASONS OF BUILDING THE ORIGINAL BASELINE

  • Baseline design avoids real-time activation scaling to limit inference computation and uses fixed-zero thresholding as a representative binarization practice.The paper describes this baseline as adding almost no floating-point computation.Source: §C.1.1 · pp. 20–21

C.1.2 SOLUTIONS OF ATTENTION WEIGHT

Alternative attention-weight thresholds address the all-one binarization problem but trade accuracy or efficiency against the original baseline. The paper motivates Bi-Attention and contrasts sign-based attention binarization with a {0,1} operator.

C.1.2 SOLUTIONS OF ATTENTION WEIGHT

  • The asymmetric threshold solution scores 48.6% average accuracy versus 50.4% for the baseline and adds 0.074G FLOPs.Its asymmetric binarized values sacrifice bitwise-operation efficiency.Source: §C.1.2 · p. 21
  • Mean-threshold binarization yields limited accuracy improvement because attention scores are asymmetric, and adds 0.076G FLOPs over the original baseline.The method's premise of symmetric floating-point values does not match the attention-score distribution.Source: §C.1.2 · p. 22
  • A 50% zero-attention setting improves average results by 1.7% for the baseline and 2.9% for BinaryBERT, while entropy-maximizing results perform best on STS-B.The ablation varies zero attention weight from 10% to 90%.Source: §C.1.2 · p. 22, Table 6 · p. 22
  • Quantile thresholds add about 0.076G (16%) FLOPs and constrain each attention tensor's entropy, limiting representation capability relative to Bi-Attention.The authors characterize the quantile approach as a more restrictive optimization problem.Source: §C.1.2 · p. 22

C.1.3 CONCLUSION

  • The paper concludes that existing attention structures are unsuitable for direct full binarization, motivating Bi-Attention for fully binarized BERT.The three alternative solutions improve accuracy only to a limited extent while reducing computational efficiency.Source: §C.1.3 · p. 22

C.2 VARIANTS OF BINARIZED ATTENTION MECHANISM

  • The sign operator maps values to {−1, 1} and cannot selectively suppress unrelated attention parts; the paper introduces bool to map values to {0, 1}.The reported comparisons support using an operator that can focus selectively on input parts and ignore others.Source: §C.2 · pp. 22–23, Table 7 · p. 23

D VISUALIZATIONS

On SST-2, the fully binarized BERT baseline loses input-dependent attention patterns, while BiBERT displays varied patterns across layers and heads.

D VISUALIZATIONS

  • The fully binarized baseline’s attention weights are identical regardless of input, unlike the selective patterns seen in the full-precision model.Figure 8 covers all heads in layers 0–3 on SST-2.Source: D VISUALIZATIONS · p. 23, Figure 8 · p. 24
  • BiBERT displays varied attention patterns across layers and heads, although some differ from full precision or appear blank.The authors attribute this to Bi-Attention’s entropy-maximizing attention-weight distribution and adaptive training, which yield input-selective attention.Source: D VISUALIZATIONS · p. 23
  • Figure 8 compares attention patterns for the full-precision model, the fully binarized baseline, and BiBERT on SST-2.The visualization includes all heads in layers 0–3.Source: Figure 8 · p. 24

Related papers

Loading 2203.06390v1…