Source-linked AI summary
ConvBERT: Improving BERT with Span-based Dynamic Convolution
Zihang Jiang, Weihao Yu, Daquan Zhou, Yunpeng Chen, Jiashi Feng, Shuicheng Yan
TL;DR
BERT-based pre-training incurs substantial computation because global self-attention is used even where some heads learn only local dependencies. ConvBERT replaces selected self-attention heads with span-based dynamic convolution and adds efficiency-oriented architectural designs; it reports consistent improvements with much less pre-training computation, including an 86.4 GLUE score for ConvBERTBASE, 0.7 above ELECTRABASE at less than one-quarter the training cost.
Problem
BERT relies heavily on global self-attention although some attention heads learn local dependencies, creating computation redundancy and high memory and computation costs.
Method
ConvBERT combines self-attention with span-based dynamic convolution and adds bottleneck attention plus grouped linear operations.
Results
ConvBERT achieves consistent performance improvements while costing much less pre-training computation; ConvBERTBASE reaches 86.4 GLUE score, 0.7 above ELECTRABASE, with less than one-quarter the training cost.
Takeaways & Limitations
The mixed design provides a more efficient backbone that captures both global and local information across downstream tasks.
Takeaways & Limitations
The dynamic convolution design is intended for local dependencies, while convolutional approaches remain limited in capturing the whole context of long sentences.
Abstract
from arXiv · showhide
Pre-trained language models like BERT and its variants have recently achieved impressive performance in various natural language understanding tasks. However, BERT heavily relies on the global self-attention block and thus suffers large memory footprint and computation cost. Although all its attention heads query on the whole input sequence for generating the attention map from a global perspective, we observe some heads only need to learn local dependencies, which means the existence of computation redundancy. We therefore propose a novel span-based dynamic convolution to replace these self-attention heads to directly model local dependencies. The novel convolution heads, together with the rest self-attention heads, form a new mixed attention block that is more efficient at both global and local context learning. We equip BERT with this mixed attention design and build a ConvBERT model. Experiments have shown that ConvBERT significantly outperforms BERT and its variants in various downstream tasks, with lower training cost and fewer model parameters. Remarkably, ConvBERTbase model achieves 86.4 GLUE score, 0.7 higher than ELECTRAbase, while using less than 1/4 training cost. Code and pre-trained models will be released.
1 Introduction
ConvBERT addresses redundancy in BERT’s self-attention by combining global self-attention with span-based dynamic convolution for local dependencies. It further adds bottleneck attention and grouped linear operations to improve efficiency while maintaining or improving performance.
- BERT’s self-attention has computation redundancy because many heads learn local dependencies and some can be removed without degrading downstream performance.
- Mixed attention integrates convolution with self-attention, combining global dependency modeling with local dependency extraction.
- Span-based dynamic convolution generates adaptive kernels from local input spans, distinguishing identical tokens that appear in different contexts.
- Bottleneck attention embeds tokens in a lower-dimensional space to reduce attention heads, while grouped linear operators reduce feed-forward parameters without hurting representation power.
- 86.4 GLUE score: ConvBERTBASE scores 5.5 higher than BERTBASE and 0.7 higher than ELECTRABASE while requiring less training cost and parameters.
2 Related work
Prior NLP models use convolution to encode local context, while pre-trained language models primarily rely on transformer self-attention. Existing convolutional approaches face difficulty capturing whole-context dependencies in long sentences, motivating combinations of convolution and transformers.
- Convolution blocks have been used in NLP to encode local information and contextual dependencies, but had not been explored in pre-training.
- Existing convolutional models are limited in capturing the whole context of long sentences, motivating sequential or multi-branch combinations with transformers.
3 Method
ConvBERT replaces selected globally computed self-attention heads with span-based dynamic convolution, combining local and global dependency modeling in a mixed-attention architecture. Bottleneck and grouped feed-forward designs further reduce computation and parameters while retaining representation power.
- Self-attention motivation: BERT self-attention computes token-pair weights globally, although many heads primarily learn local dependencies, creating unnecessary computation and redundancy.
- Dynamic convolution: Dynamic convolution has linear sequence-length complexity and models local dependencies efficiently, but kernels conditioned on single tokens ignore contextual differences.
- Span-based dynamic convolution: Span-based dynamic convolution gathers a token span before generating kernels, allowing identical tokens in different contexts to receive different kernels.
- Mixed attention: Mixed attention concatenates self-attention and span-based dynamic convolution outputs; both share queries but use different keys to generate attention maps and convolution kernels.
- ConvBERT architecture: The bottleneck design projects embeddings to dimension d/γ and reduces attention heads by γ, saving self-attention computation while encouraging more compact attention information.
- ConvBERT architecture: Grouped linear feed-forward layers process feature groups independently and concatenate them, improving efficiency with negligible performance drop; stacking these modules forms ConvBERT.
4 Experiment
The experiments evaluate ConvBERT across pre-training settings, ablations, GLUE, and SQuAD, showing efficient performance gains from its architectural components.
- Experimental setup: ConvBERT is evaluated with replaced token detection pre-training across multiple model sizes and training durations.Small- and base-sized models use sequence length 128 and are evaluated after 1M and 4M updates.
- Evaluation: The study evaluates models on GLUE classification tasks and SQuAD question answering using standard task-specific metrics.GLUE score averages eight tasks, while SQuAD reports Exact Match and F1 on development sets.
- Ablation study: Bottleneck structure and grouped linear operations reduce parameters and computation cost without substantially hurting performance.The authors suggest the remaining attention heads learn more compact representations that may generalize better.
- Ablation study: Larger convolution kernels improve results until their receptive field covers the full sentence, after which additional benefit diminishes.Kernel size 9 is used in later experiments because it gives the best result.
- Ablation study: Span-based dynamic convolution improves GLUE performance more than conventional depth-wise separable or ordinary dynamic convolution integrations.Directly adding conventional convolution hurts performance, while dynamic convolution provides little improvement over baseline BERT.
- Comparison results: On GLUE, ConvBERT small and base models outperform similarly sized baselines while requiring much less pre-training cost.ConvBERTBASE performs better than ELECTRABASE with less than one-quarter of the pre-training cost; large-model comparison is deferred.
- Comparison results: On SQuAD, ConvBERT small models outperform ELECTRASMALL and are comparable to BERTBASE, while the base model outperforms similarly sized models.The comparison accounts for the lower pre-training cost of ConvBERT and notes that MobileBERT uses knowledge distillation and development-set search.
5 Conclusion
The paper concludes that span-based dynamic convolution, mixed attention, bottleneck structure, and grouped linear operations improve ConvBERT performance while reducing pre-training computation.
- Conclusion: ConvBERT combines span-based dynamic convolution with self-attention in a mixed attention block for language pre-training.The architecture also includes a bottleneck structure and grouped linear operation.
- Conclusion: Experiments report consistent performance improvements with substantially lower pre-training computation.
Broader impact
ConvBERT is presented as a smaller, more efficient pre-trained language model with lower training cost. Its efficiency may also increase privacy risks when used to process personal social-media text.
- ConvBERT offers smaller model size and lower training cost than BERT.
- Its efficiency could support detecting and understanding personal text posts on social platforms, but may bring privacy threats.
6 Appendix
The appendix describes the GLUE and SQuAD evaluation resources used in the study. GLUE covers diverse language-understanding tasks, while SQuAD evaluates reading comprehension through answer spans, including unanswerable questions in version 2.0.
- GLUE is a collection of nine natural language understanding tasks evaluated through a held-out testing server.
- MNLI tests entailment relations across premise–hypothesis pairs drawn from ten genres of written and spoken English.
- QNLI is a binary task asking whether a context sentence contains the answer to a question.
- QQP classifies whether pairs of Quora questions are semantically equivalent.
- RTE predicts entailment versus non-entailment using data from multiple textual-entailment challenges.
- SST-2 performs binary sentiment classification on movie-review sentences.
- MRPC evaluates semantic equivalence between sentence pairs from online news.
- CoLA is a binary classification task for judging whether English sentences are grammatical.
6.2 Pre-training details
ConvBERT uses replaced token detection for pre-training, with a small masked-language-model generator producing replaced-token examples. The configuration largely follows ELECTRA, with longer sequences added before SQuAD fine-tuning.
- Replaced token detection trains the model to predict whether each sequence token was replaced.
- A small generator trained with masked language modelling creates the replaced-token training examples.
- The pre-training configuration mostly follows ELECTRA's hyper-parameters.
- 512-token inputs are used for 10% more updates before fine-tuning on longer-context tasks such as SQuAD.
- Table 5 lists pre-training hyper-parameters and specifies Adam as the optimizer.
6.3 Fine-tuning details
Fine-tuning searches over learning rates, weight decay, and training epochs, while additional GLUE experiments test model sizes and pre-training settings. ConvBERT consistently outperforms the original BERT architecture across those settings.
- Learning rates, weight decay, and training epochs are selected by grid search, with other parameters matching ELECTRA.
- ConvBERT consistently outperforms the original BERT architecture across model sizes, pre-training tasks, and dataset sizes.
6.5 More examples and analysis of attention map
The section supplements attention-map examples with a diagonal-concentration metric and reports that BERT attention focuses more on local dependencies on MRPC.
- Additional attention-map examples are provided in Figure 6.
- Diagonal concentration is introduced as a quantitative measure of how much local dependency an attention map captures.
- Table 7 compares ConvBERT with transformer-based BERT across model sizes and pre-training settings using GLUE score, MLM, and RTD.
- BERT attention concentrates more on local dependency according to the MRPC results in Table 8.
6.6 Inference speed
On Intel CPU, the mixed-attention block uses fewer floating-point operations and runs faster than self-attention. GPU and TPU implementation remains insufficiently optimized, limiting acceleration in some settings.
- The mixed-attention block has lower Flops and is faster than self-attention on an Intel CPU.
- GPU and TPU implementations are not yet well optimized for efficiency.
- Acceleration may not be obvious when input sequences are short.
- Inference speed is reported in Table 9.