PaperlayerView exact revision on arXiv

Source-linked AI summary

SqueezeBERT: What can computer vision teach NLP about efficient neural networks?

Forrest N. Iandola, Albert E. Shaw, Ravi Krishna, Kurt W. Keutzer

arXiv:2006.11316v1cs.CLcs.CVcs.LG

TL;DR

Accurate NLP models such as BERT can be too computationally expensive for practical mobile deployment. SqueezeBERT applies grouped convolutions to self-attention layers, and reports more than a 4x latency decrease over BERT-base with competitive accuracy.

  • Problem

    Accurate NLP models such as BERT are computationally expensive for mobile devices; BERT-base takes 1.7 seconds to classify a sample on a Pixel 3.

    Source: §1 · p. 2

  • Method

    SqueezeBERT applies convolutions, particularly grouped convolutions, to layers in a self-attention network for NLP.

    Source: §1.3 · p. 2, §2.3 · p. 4

  • Results

    More than 4x latency decrease over BERT-base on a Pixel 3, with accuracy competitive with distillation-trained MobileBERT and the original BERT-base.

    Source: §7 · p. 8

  • Takeaways & Limitations

    SqueezeBERT offers a speed-accuracy tradeoff the authors consider compelling for mobile NLP inference.

    Source: §5.2 · p. 8

    The authors report using standard datasets and are unaware of experimental factors that would change SqueezeBERT’s bias propensity relative to similar self-attention research.

    Source: Potential effects of unintended bias… · p. 9

Abstract

from arXiv · show

Humans read and write hundreds of billions of messages every day. Further, due to the availability of large datasets, large computing systems, and better neural network models, natural language processing (NLP) technology has made significant strides in understanding, proofreading, and organizing these messages. Thus, there is a significant opportunity to deploy NLP in myriad applications to help web users, social networks, and businesses. In particular, we consider smartphones and other mobile devices as crucial platforms for deploying NLP models at scale. However, today's highly-accurate NLP neural network models such as BERT and RoBERTa are extremely computationally expensive, with BERT-base taking 1.7 seconds to classify a text snippet on a Pixel 3 smartphone. In this work, we observe that methods such as grouped convolutions have yielded significant speedups for computer vision networks, but many of these techniques have not been adopted by NLP neural network designers. We demonstrate how to replace several operations in self-attention layers with grouped convolutions, and we use this technique in a novel network architecture called SqueezeBERT, which runs 4.3x faster than BERT-base on the Pixel 3 while achieving competitive accuracy on the GLUE test set. The SqueezeBERT code will be released.

1 Introduction and Motivation

Mobile devices are an important setting for NLP, but accurate self-attention models can be costly to run there. SqueezeBERT applies computer-vision convolution techniques to NLP and reports lower smartphone latency with competitive accuracy.

1 Introduction and Motivation

  • NLP could support mobile users with writing assistance, content moderation, message classification, feed composition, prioritization, and duplicate detection.The paper situates these uses against the scale of daily messaging and mobile access.Source: §1 · p. 1
  • BERT-base takes approximately 1.7 seconds to classify one text sample on a Google Pixel 3, making mobile deployment a practical concern.The introduction points to mobile use for reading, writing, and sharing text.Source: §1 · p. 2

1.1 What has CV research already taught NLP research about efficient networks?

  • MobileBERT brings two computer-vision ideas to NLP: bottleneck layers that reduce channels before attention and residual connections across high-channel-count layers.The paper also notes MobileBERT’s approximately 0.6-second Pixel 3 classification time and higher GLUE accuracy than several efficient networks.Source: §1.1 · p. 2

1.2 What else can CV research teach NLP research about efficient networks?

  • Grouped convolutions are widely used in efficient computer-vision networks, while the authors report they were unaware of prior NLP work applying them.The paper identifies convolutions and grouped convolutions as additional computer-vision ideas to apply to NLP.Source: §1.2 · p. 2

1.3 SqueezeBERT: Applying lessons learned from CV to NLP

  • SqueezeBERT runs at lower smartphone latency than BERT-base, MobileBERT, and several efficient NLP models while maintaining competitive accuracy.The paper introduces it as a self-attention network applying convolutions, particularly grouped convolutions.Source: §1.3 · pp. 2–3

2 Implementing self-attention with convolutions

The section identifies position-wise fully-connected layers as the main source of BERT’s measured latency. It shows that these layers can be implemented as equivalent kernel-size-1 convolutions, motivating convolution-based acceleration.

2.1 Self-attention networks

  • Each BERT encoder block has a self-attention module followed by three feed-forward position-wise fully-connected layers, while attention uses separate layers to generate Q, K, and V.These layers apply the same operation independently at each feature position.Source: §2.1 · p. 3

2.2 Benchmarking BERT for mobile inference

  • PFC layers account for 88.3% of total latency: 18.9% in self-attention modules and 69.4% in feed-forward network modules.Source: §2.2 · p. 4

2.3 Replacing the position-wise fully connected layers (PFC) with convolutions

  • A position-wise fully-connected layer is equivalent to a 1D convolution with kernel size 1, preserving the network’s numerical properties and behavior.This equivalence supports implementing the costly layers with convolutions.Source: §2.3 · p. 4

3 Incorporating grouped convolutions into self-attention

Grouped convolutions reduce the weights and FLOPs needed for convolutional layers. SqueezeBERT uses them in most of its self-attention and feed-forward layers while retaining the BERT-base architecture’s other stated aspects.

3 Incorporating grouped convolutions into self-attention

  • A grouped 1D convolution uses 1/G as many FLOPs and weights as an ordinary convolution, excluding the unchanged channel-wise bias operations.It divides the input into G groups processed with independent weights.Source: §3 · p. 4

3.1 SqueezeBERT

  • SqueezeBERT uses G = 4 in self-attention PFC layers and FFN2/FFN3, but G = 1 in FFN1 to allow mixing across groups.With G = 4, all FFN layers have the same number of arithmetic operations; the architecture otherwise adopts BERT-base’s stated configuration.Source: §3.1 · pp. 4–5

4 Experimental Methodology

The experiments use Wikipedia and BooksCorpus for pretraining and GLUE for evaluation, separating a simple training protocol from additional training techniques. The latter combines final-layer distillation with transfer learning from MNLI for most GLUE tasks.

4.1 Datasets

  • Pretraining uses Wikipedia and BooksCorpus, while GLUE supplies nine diverse NLU tasks for finetuning and evaluation.Three percent of the combined pretraining data is held out as a test set; GLUE performance is treated as an approximation of generalizability, especially to other text-classification tasks.Source: §4.1 · p. 5

4.2 Training Methodology

  • The evaluation separates a simple training scheme from a second scheme with distillation and other techniques to distinguish model contributions from training-scheme contributions.The authors note that training schemes vary across prior work, complicating comparisons of final accuracy.Source: §4.2 · p. 5

4.2.1 Training without bells and whistles

  • Without additional training techniques, SqueezeBERT is pretrained from scratch with LAMB and finetuned with AdamW, while BERT-base follows the same protocol for comparison.Pretraining runs for 56k steps at sequence length 128, then 6k steps at length 512; finetuning tunes learning rate and dropout for each task.Source: §4.2.1 · p. 5

4.2.2 Training with bells and whistles

  • With additional techniques, SqueezeBERT uses final-layer distillation only during finetuning and transfers MNLI-trained student weights to other GLUE tasks except CoLA.The teacher is ELECTRA-pretrained BERT-base finetuned on MNLI and then independently on each task; prior work uses varied distillation layers and training stages [15] [17] [21] [22].Source: §4.2.2 · pp. 5–6

5 Results

With additional training techniques, SqueezeBERT's average GLUE score is within 0.4 percentage points of MobileBERT, and the paper describes both as offering a compelling speed-accuracy tradeoff for mobile inference. Without those techniques, SqueezeBERT is 4.3x faster than BERT-base and exceeds MobileBERT's accuracy on the four tasks where both report results.

5.1 Results without bells and whistles

  • 4.3x faster than BERT-base, SqueezeBERT also exceeds MobileBERT's accuracy on all four tasks where both report results without distillation.These are development-set comparisons; MobileBERT is 3.0x faster than BERT-base in the authors' measurements.Source: §5.1 · p. 7
  • Without additional training techniques, ALBERT-base has 9x fewer parameters than BERT-base but the same FLOPs and no measured smartphone speedup.On the two GLUE tasks with reported ALBERT-base accuracy, SqueezeBERT and MobileBERT both outperform it.Source: §5.1 · p. 7

5 Results

  • The comparisons include prior results with incomplete GLUE task coverage, and all latency values in the results tables were benchmarked by the authors.The authors use their 572ms Pixel 3 measurement for MobileBERT rather than the 620ms reported by its authors.Source: §5.2 · p. 7

5.2 Results with bells and whistles

  • Within 0.4 percentage points of MobileBERT on average across GLUE, SqueezeBERT offers a compelling mobile speed-accuracy tradeoff with the additional training techniques.The comparison is reported for models trained with bells and whistles; Table 3 covers GLUE test-set results.Source: Table 3 · p. 7, §5.2 · p. 8

6 Related Work

The related work describes quantization and pruning as potential efficiency techniques for SqueezeBERT, while noting that they are outside this paper's focus. It also distinguishes prior uses of convolutions in language generation from the paper's NLU emphasis.

Quantization and Pruning

  • Quantization and pruning could further improve SqueezeBERT's efficiency, but neither is a focus of this paper.Quantization reduces parameter or activation bit width while aiming to maintain accuracy [47]; pruning eliminates parameters to reduce storage and potentially computation [48].Source: §6 · p. 8

Convolutions in self-attention networks for language-generation tasks

  • Prior convolution work includes architectures combining convolutions and self-attention for language generation, convolution-only models competitive on some generation tasks, and sparsified attention inspired by dilated convolutions.The paper's experiments instead focus on NLU tasks such as sentence classification; earlier work also applied convolutional networks to NLU.Source: §6 · p. 8

7 Conclusions & Future Work

The paper applies grouped convolutions from computer vision to NLP and presents SqueezeBERT as an efficient self-attention model. It identifies sequence downsampling and automated architecture search as future directions.

7. Conclusions & Future Work

  • SqueezeBERT implements most self-attention encoder layers with 1D grouped convolutions, yielding a greater-than-4x latency decrease over BERT-base on Pixel 3 while achieving competitive accuracy after distillation.The paper also shows that position-wise fully connected layers can be implemented with mathematically equivalent 1D convolutions.Source: §7 · p. 8
  • Future directions include downsampling sequence length across layers, changing hidden sizes, and exploring U-Nets, BiFPNs, striding, and dilation.These possibilities are presented as CV techniques not covered in the paper that may be applied to NLP.Source: §7 · p. 8
  • The expanded space of possible NLP architectures motivates automated neural architecture search approaches.Source: §7 · p. 8

Broader Impact

Mobile NLP may support real-time and offline text applications, but the authors also warn that freely released models could be used for surveillance or censorship. They note unresolved questions about bias in the datasets used.

Potential benefits of this work

  • Mobile inference may avoid expensive server infrastructure and support privacy-sensitive, real-time, or offline uses such as grammar checking and sentence completion.Other examples include email sorting, chat analysis, spam detection, and hate-speech filtering.Source: Potential benefits of this work · p. 9

Potential malicious uses of this work

  • Free software that can be trained on any dataset could also support smart key-loggers, eavesdropping viruses, or politically motivated content censorship.The authors note that local computation could help malicious software avoid detection by uploading only selected information.Source: Potential malicious uses of this work · p. 9
  • The authors report no known experimental factors that would alter SqueezeBERT’s bias propensity relative to BERT and ELECTRA, while citing prior studies of gender bias.Their pretraining corpora are Wikipedia and BooksCorpus; the cited studies investigate their gender-related biases and biases in BERT and GPT models finetuned on GLUE.Source: Potential effects of unintended bias… · p. 9
  • Aside from gender bias, the authors know of no studies examining other bias patterns in their datasets, though they cite GPT-2 and GPT-3 studies [64].They point readers to Blodgett et al. [65] and Sun et al. for bias-minimization suggestions.Source: Potential effects of unintended bias… · p. 9
  • The cited taxonomy distinguishes allocational harms, such as unfairly distributing opportunities, from representational harms, such as demeaning or erasing social groups [65].The paper presents these as harms caused by biased NLP technology.Source: Potential effects of unintended bias… · p. 9

A Overview of GLUE tasks

This overview describes the GLUE tasks, their metrics, and possible applications. It also explains how task scores are aggregated and flags metric-reporting differences that affect comparisons.

A Overview of GLUE tasks

  • MNLI tests sentence entailment, RTE classifies entailment, and QNLI checks whether a sentence contains an answer to a question.Their reported metrics are the mean of MNLI-m and MNLI-mm accuracy, RTE accuracy, and QNLI accuracy, respectively; applications include writing checks, inference, and question answering.Source: A Overview of GLUE tasks · pp. 15–16
  • QQP and MRPC assess whether question pairs or sentence pairs have the same meaning, using accuracy and F1 averages for both tasks.The overview gives organizing similar questions as a QQP application and plagiarism detection as an MRPC application.Source: A Overview of GLUE tasks · pp. 15–16
  • The overview cautions that QQP test scores are not comparable when papers report different metrics, so it omits TinyBERT and ELECTRA-small results reported on only one metric.The paper reports BERT-base QQP F1=71.2 and accuracy=89.2 as an example of the gap between the two metrics.Source: A Overview of GLUE tasks · p. 15
  • SST-2 classifies sentiment, while CoLA judges whether English sentences are acceptable; their metrics are accuracy and MCC, respectively.The overview suggests helpdesk triage and chatbot support for SST-2, and grammatical feedback for CoLA.Source: A Overview of GLUE tasks · pp. 15–16
  • STS-B predicts sentence similarity on a 1-to-5 scale, while WNLI predicts a pronoun’s antecedent; the paper uses majority-class prediction for WNLI, yielding 65.1% test accuracy.STS-B is a regression task scored by the mean of Pearson and Spearman correlations; WNLI uses accuracy.Source: A Overview of GLUE tasks · p. 16
  • GLUE scores follow leaderboard conventions, averaging task metrics and treating MNLI-m and MNLI-mm as one task; WNLI is omitted from the development-set score.The overview notes that tasks may have one or two metrics, including accuracy, F1, MCC, Pearson, and Spearman correlation.Source: A Overview of GLUE tasks · pp. 15–16

B.1 Training Hardware

The appendix documents the computing resources and software used for training, estimates cloud costs, and specifies the distillation target and training setup.

B.1 Training Hardware

  • The project used two 8-GPU servers for about six months, which the authors estimate would have cost around $100,000 to rent in the cloud.They report using their own resources rather than cloud or corporate computing resources.Source: §B.1 · p. 17
  • Reproducing SqueezeBERT from scratch is estimated to take about five days and cost approximately $2,880 on one 8-V100 AWS machine.The estimate includes four days of pretraining and under one day of finetuning across GLUE tasks.Source: §B.1 · p. 17

B.2 Training Software

  • Pretraining used eight GPUs with 16-bit math, while finetuning used one GPU with 32-bit math and concurrent tasks across the server.The code is PyTorch-based and uses HuggingFace Transformers, NVIDIA Deep Learning Examples, and NVIDIA Apex mixed precision.Source: §B.2 · p. 17

B.3 Further details: distillation

  • Distillation blends teacher logits with one-hot ground truth as the soft cross-entropy target, with α controlling their relative weighting.The authors tune α during hyperparameter search.Source: §B.3 · p. 17

B.4 Details of hyperparameter search during finetuning on GLUE tasks

The appendix reports hyperparameter-search observations, specifies the smartphone latency protocol, and situates SqueezeBERT among computer-vision and NLP efficiency techniques.

B.4 Details of hyperparameter search during finetuning on GLUE tasks

  • The authors grid-searched finetuning hyperparameters, and search breadth and epoch count varied by task size.Smaller datasets such as RTE and MRPC received broader searches and more epochs; larger datasets such as MNLI and QQP received narrower searches and fewer epochs.Source: §B.4 · pp. 17–18, Table 4 · p. 18
  • On three of eight GLUE tasks—MNLI, QNLI, and CoLA—distillation did not outperform finetuning without it; QNLI reached its peak development accuracy after two epochs.The authors also note that the best development-set accuracy was not necessarily achieved at the final epoch.Source: §B.4 · p. 18

C Further details: Inference on a smartphone

  • Smartphone latency was measured on a Pixel 3 using CPU cores, length-128 inputs, batch size one, and average latency over 40 runs.Networks were exported to TorchScript and run using a modified PyTorch benchmark.Source: C Further details: Inference on a… · p. 18

D More related work

  • From 2016 to 2020, computer-vision systems reduced FLOPs while improving accuracy: by 20x on ImageNet, 38x on COCO detection, and 334x on Cityscapes segmentation.The cited examples compare ResNet-101 with FixEfficientNet-D0, FPN with EfficientDet-D1, and FCN-8s with SqueezeNAS-MAC-Small, respectively.Source: D More related work · pp. 18–19
  • Grouped convolutions are a prominent design element in recent efficient vision networks, reducing computation and parameters through channel-wise structured sparsity.The paper notes their use in FixEfficientNet, EfficientDet, and SqueezeNAS and cites MobileNet’s accuracy-preserving results in some settings [31].Source: D More related work · p. 19
  • Improved pretraining and training regimes can raise sentence-classification accuracy without changing the BERT encoder design, as reported for RoBERTa and ELECTRA [13].The authors suggest that more data, more iterations, or adversarial pretraining might further improve SqueezeBERT.Source: D More related work · p. 19
  • MT-DNN amortizes neural-network computation across sentence-classification tasks, an approach the authors identify as a possible avenue for future speedups.They note this could be explored for tasks such as sentiment analysis and linguistic correctness checking.Source: D More related work · p. 19
Loading 2006.11316v1…