PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantized Transformer Language Model Implementations on Edge Devices

Mohammad Wali Ur Rahman, Murad Mehrab Abrar, Hunter Gibbons Copening, Salim Hariri, Sicong Shao, Pratik Satam, Soheil Salehi

arXiv:2310.03971v1cs.CLcs.AR

TL;DR

Large transformer models are difficult to deploy on resource-constrained devices because of their size and inference demands. This study evaluates converted and quantized MobileBERT models on three edge devices for tweet reputation analysis, finding substantially smaller models with modest accuracy loss and at least one prediction per second.

Abstract

from arXiv · show

Large-scale transformer-based models like the Bidirectional Encoder Representations from Transformers (BERT) are widely used for Natural Language Processing (NLP) applications, wherein these models are initially pre-trained with a large corpus with millions of parameters and then fine-tuned for a downstream NLP task. One of the major limitations of these large-scale models is that they cannot be deployed on resource-constrained devices due to their large model size and increased inference latency. In order to overcome these limitations, such large-scale models can be converted to an optimized FlatBuffer format, tailored for deployment on resource-constrained edge devices. Herein, we evaluate the performance of such FlatBuffer transformed MobileBERT models on three different edge devices, fine-tuned for Reputation analysis of English language tweets in the RepLab 2013 dataset. In addition, this study encompassed an evaluation of the deployed models, wherein their latency, performance, and resource efficiency were meticulously assessed. Our experiment results show that, compared to the original BERT large model, the converted and quantized MobileBERT models have 160$\times$ smaller footprints for a 4.1% drop in accuracy while analyzing at least one tweet per second on edge devices. Furthermore, our study highlights the privacy-preserving aspect of TinyML systems as all data is processed locally within a serverless environment.

I. INTRODUCTION

The paper addresses the difficulty of deploying large transformer NLP models on resource-constrained edge devices by evaluating MobileBERT with TinyML and TensorFlow-Lite techniques. It compares model performance, resource use, and privacy-preserving local processing for reputation-polarity classification.

I. INTRODUCTION

  • Large transformer NLP models require substantial storage, memory, computation, and latency, making real-time deployment on constrained edge devices difficult.TinyML is presented as an approach for deploying machine-learning models on small, low-power devices.Source: §I · p. 1
  • TinyML processes data locally at the device or edge node, reducing communication delays and privacy risks associated with remote-server analysis.Local processing can support faster decision-making without relying on network availability or quality.Source: §I · p. 1
  • The proposed evaluation framework deploys TensorFlow-Lite MobileBERT models on resource-constrained devices with and without quantization.The framework is designed to assess performance and resource usage while examining the trade-off between model size, latency, and accuracy.Source: §I · p. 2
  • The system is designed to process data locally on edge devices, providing a low-cost and privacy-preserving TinyML implementation.The stated contribution emphasizes local processing rather than communication with a central server.Source: §I · p. 2
  • The study evaluates MobileBERT for classifying social-media text by reputation polarity and compares it with BERT Large.The models are fine-tuned for the specific NLP task under study.Source: §I · p. 2

II. RELATED WORKS

Prior work has reduced neural-model size and computation through compression, quantization, hardware-aware optimization, and compact transformer architectures. The paper positions its contribution as a more comprehensive resource-utilization analysis of TensorFlow-Lite MobileBERT implementations on constrained devices.

II. RELATED WORKS

  • Earlier studies used pruning, trained quantization, Huffman coding, and integer-only inference to reduce model size or computation while preserving accuracy.These approaches established compression and quantization as strategies for resource-constrained deployment.Source: §II · p. 2
  • Hardware-aware methods such as HAQ and AMC search for compression settings that balance model performance with hardware resource constraints.The described methods use reinforcement learning or hardware-aware objectives to explore quantization and compression choices.Source: §II · p. 2
  • Compact NLP architectures and parameter-efficient methods reduce attention, computation, or parameter requirements while targeting performance comparable to full-precision models.Examples include adapter-based transfer learning, LITE Transformer, and Micronet.Source: §II · p. 2
  • TinyBERT and MobileBERT are compact BERT variants for resource-constrained applications, and prior work also evaluated MobileBERT in TensorFlow-Lite on a smartphone.The cited smartphone study considered question-answering and text-generation tasks.Source: §II · p. 2
  • The paper identifies incomplete comprehensive analysis of resource utilization in prior TensorFlow-Lite MobileBERT studies as the gap it addresses.Its stated contribution is an evaluation of MobileBERT implementations on resource-constrained devices.Source: §II · p. 2, Figure 1 · p. 3

III. FRAMEWORK ARCHITECTURE

The framework collects and preprocesses reputation-related tweets, fine-tunes BERT Large and MobileBERT, converts MobileBERT to TensorFlow-Lite, and evaluates deployment performance and resource use locally on edge devices.

III. FRAMEWORK ARCHITECTURE

  • The framework collects and preprocesses data, fine-tunes the models, converts MobileBERT to TensorFlow-Lite, applies quantization, and analyzes deployed-model performance and resource use.All processing occurs locally on the edge device without communication with a central server.Source: §III · p. 3

A. Data Collection and Preprocessing

  • The experiment fine-tunes BERT Large and MobileBERT on RepLab 2013 for multiclass classification based on reputation polarity.RepLab 2013 contains tweets about 61 entities from four domains.Source: A. Data Collection and Preprocessing · p. 3
  • The study uses English tweets because MobileBERT is not optimized for multilingual tasks.English tweets are selected from the RepLab 2013 dataset for the fine-tuning process.Source: A. Data Collection and Preprocessing · p. 3
  • Preprocessing removes redundant spaces, symbols, emojis, URLs, and punctuation before tokenizing tweets with the appropriate BERT tokenizer.The cleaning step prepares social-media text for model training.Source: A. Data Collection and Preprocessing · p. 3

B. Fine-tuning of the Pre-trained Models

The study fine-tunes BERT Large and MobileBERT for reputation polarity classification, then compares their performance and resource demands for edge deployment.

B. Fine-tuning of the Pre-trained Models

  • BERT Large, with 340 million parameters, is selected as the baseline for evaluating MobileBERT on resource-constrained edge devices.MobileBERT derives from IB-BERT through knowledge distillation and is optimized for edge use.Source: B. Fine-tuning of the Pre-trained Models · pp. 3–4
  • Both models represent each token using positional, sentence, and word embeddings, with [CLS] and [SEP] tokens added to structure the input.For reputation polarity, the final hidden state of [CLS] summarizes the input sequence.Source: B. Fine-tuning of the Pre-trained Models · p. 4
  • A softmax classifier uses the [CLS] hidden state to predict reputation polarity classes, while fine-tuning jointly optimizes BERT and classifier parameters.The task-specific classifier is parameterized by matrix V.Source: B. Fine-tuning of the Pre-trained Models · p. 4
  • After training, BERT Large remains a TensorFlow model for workstation evaluation, whereas MobileBERT is converted and compressed for edge-device deployment.This establishes the comparison between the desktop baseline and optimized edge model.Source: B. Fine-tuning of the Pre-trained Models · p. 4

C. Model Compression and Optimization

The framework converts trained MobileBERT models to TensorFlow-Lite FlatBuffers, applies dynamic-range quantization, and deploys model variants across desktop and embedded devices for comparative evaluation.

C. Model Compression and Optimization

  • MobileBERT is exported, frozen, converted to a TensorFlow-Lite FlatBuffer, and then optimized with dynamic-range quantization.Dynamic-range quantization converts weights and activations from floating point to 8-bit integers using data-dependent ranges.Source: C. Model Compression and Optimization · p. 4, Figure 2 · p. 4

D. Model Deployment

  • Hardware specifications and experimental results are organized in Table II and Table III for comparing device and model behavior.The supplied passages identify these tables but do not provide their cell values.Source: Table II · p. 4, Table III · p. 5

C. Model Compression and Optimization

  • Dynamic-range quantization reduces model size and computational requirements while aiming to avoid significant performance loss.The paper contrasts this with fixed-range quantization, which can cause information loss and larger performance degradation.Source: C. Model Compression and Optimization · p. 4

D. Model Deployment

  • The study deploys quantized and non-quantized TensorFlow-Lite models on Linux-based embedded devices, while TensorFlow models run on a desktop CPU.Performance and resource-utilization data are collected during inference for comparative analysis.Source: D. Model Deployment · p. 4
  • The TensorFlow-Lite interpreter loads converted models into device memory, executes their graphs, and exposes APIs for inputs and outputs.Figure 2 represents the conversion and deployment path from TensorFlow models to target machines.Source: D. Model Deployment · p. 4, Figure 2 · p. 4

E. Model Evaluation and Analysis

The evaluation uses complementary metrics to assess deployed models’ latency, computational efficiency, performance, and resource efficiency.

E. Model Evaluation and Analysis

  • The study introduces Speed Index, Model Performance Index, and Resource Efficiency Ratio to evaluate latency, performance, and efficiency together.These metrics are intended to provide a holistic assessment of deployed model effectiveness.Source: E. Model Evaluation and Analysis · p. 5

1) Speed Index (SI):

  • Speed Index captures the trade-off among floating-point operations per second, quantization bits, and total inference time.Its equation uses Q for quantization bits and t for total time in seconds.Source: 1) Speed Index (SI) · p. 5
  • The Speed Index quantifies processing speed relative to the model’s operation count, quantization level, and elapsed time.The metric is designed to reflect computational efficiency alongside speed.Source: 1) Speed Index (SI) · p. 5

2) Model Performance Index (MPI):

The paper supplements speed evaluation with composite metrics that combine predictive performance, power dissipation, and resource utilization.

2) Model Performance Index (MPI):

  • Model Performance Index combines average accuracy, average F-Score, and total power dissipation to assess overall performance and energy efficiency.Higher MPI values indicate better overall performance under this balance.Source: 2) Model Performance Index (MPI) · p. 5

3) Resource Efficiency Ratio (RER):

  • Resource Efficiency Ratio measures utilization efficiency using total energy consumption, CPU utilization, and memory utilization.The metric is intended to quantify how efficiently a model uses device resources.Source: 3) Resource Efficiency Ratio (RER) · p. 5

IV. EXPERIMENT RESULTS

The experiment section defines the evaluation protocol and reports reproducibility procedures for assessing model performance and resource use.

IV. EXPERIMENT RESULTS

  • The section presents experimental settings and a comprehensive discussion of the results.Source: §IV · p. 5

A. Experiment Settings

  • Model performance was evaluated using Accuracy and F-score metrics, consistent with RepLab 2013, and both models used a learning rate of 1×10−5.Source: A. Experiment Settings · p. 5
  • Five random seeds were used on all devices during each experiment iteration to support reproducibility.Source: A. Experiment Settings · p. 5

1) CPU and Memory Utilization Data using PSUTIL:

The study measures system and process resource use with PSUTIL and estimates Raspberry Pi current through an ACS712-based sensing setup.

1) CPU and Memory Utilization Data using PSUTIL:

  • PSUTIL collects system-wide and process-specific CPU and memory utilization before, during, and after model execution.Experiments start simultaneously, although models finish at different times because their latencies vary.Source: 1) CPU and Memory Utilization Data… · p. 5
  • Model deployments span desktop TensorFlow versions and 32-, 16-, and 8-bit TensorFlow-Lite MobileBERT versions on Raspberry Pi 3B, 3B+, and 4B.The evaluated model footprints range from 4GB for BERT Large to 25MB for 8-bit quantized MobileBERT.Source: 1) CPU and Memory Utilization Data… · p. 5

2) Current Sensing using ACS712 Sensor:

B. Results from Comparative Analysis

The comparative analysis evaluates deployed models across latency, resource use, power, accuracy, F-score, and efficiency metrics on Raspberry Pi devices.

B. Results from Comparative Analysis

  • At least one prediction per second was achieved by every 8-bit quantized TensorFlow-Lite model, while the fastest was 1.15× slower than BERT Large with a 160× smaller footprint.Non-quantized models on Raspberry Pi 3B and 3B+ were 5.55× and 4.76× slower than BERT Large, respectively.Source: Figure 6 · p. 6, B. Results from Comparative Analysis · p. 7
  • TensorFlow-Lite models used significantly fewer resources, while Raspberry Pi 3B showed higher system-wide CPU usage and latency associated with paging delays.Source: B. Results from Comparative Analysis · p. 6, Figure 7 · p. 7
  • Raspberry Pi 3B generally required less power but had longer inference times than Raspberry Pi 3B+ and Raspberry Pi 4B.Raspberry Pi 4B had the highest FLOPS across quantization options, while accuracy and F-score remained comparable with slight quantization-based variation.Source: B. Results from Comparative Analysis · p. 7
  • Raspberry Pi 4B with 8-bit quantization achieved the strongest SI and MPI values, whereas Raspberry Pi 3B+ configurations showed high RER values.These metrics support device and quantization selection based on speed, performance, and resource-efficiency trade-offs.Source: B. Results from Comparative Analysis · p. 7

V. CONCLUSION

The paper demonstrates TinyML-based NLP on edge devices by fine-tuning BERT models and evaluating quantized TensorFlow-Lite MobileBERT deployments. These models support reputation-polarity analysis with lower resource use and smaller footprints than BERT Large.

V. CONCLUSION

  • The experiments apply TinyML to NLP by fine-tuning BERT models for identifying reputation polarity from text.Source: §V · pp. 7–8
  • Quantized TensorFlow-Lite MobileBERT models achieve performance comparable to BERT Large with significantly lower resource utilization and smaller code footprints.The evaluation focuses on deploying these models on Raspberry Pi devices for reputation-polarity analysis.Source: §V · p. 8
  • The study evaluates MobileBERT TensorFlow-Lite models on Raspberry Pi devices, addressing deployment on hardware with fewer resources.Source: §V · p. 8
  • Future work will investigate federated learning for training models on resource-constrained edge devices while preserving privacy.Source: §V · p. 8

Related papers

Loading 2310.03971v1…