Source-linked AI summary
Comparing BERT against traditional machine learning text classification
Santiago González-Carvajal, Eduardo C. Garrido-Merchán
TL;DR
The paper asks whether BERT should be preferred by default over traditional TF-IDF-based machine-learning approaches for NLP text classification. It compares the methods across four scenarios, including different languages, and reports that BERT outperformed the traditional approach while being less complicated to implement. The authors also note that BERT’s results can still be improved.
Problem
The paper examines whether empirical evidence supports using BERT as a default NLP technique instead of traditional TF-IDF-based methods, including under multilingual conditions.
Method
The study compares BERT with traditional machine-learning classifiers trained on TF-IDF features across four text-classification experiments and languages.
Results
BERT outperformed the traditional NLP approach across four scenarios, including different languages, and was less complicated to implement.
Takeaways & Limitations
The findings add empirical evidence supporting BERT as a default technique for average NLP problems.
Takeaways & Limitations
The authors acknowledge that BERT’s results can be improved and identify hyperparameter auto-tuning as future work.
Abstract
from arXiv · showhide
The BERT model has arisen as a popular state-of-the-art machine learning model in the recent years that is able to cope with multiple NLP tasks such as supervised text classification without human supervision. Its flexibility to cope with any type of corpus delivering great results has make this approach very popular not only in academia but also in the industry. Although, there are lots of different approaches that have been used throughout the years with success. In this work, we first present BERT and include a little review on classical NLP approaches. Then, we empirically test with a suite of experiments dealing different scenarios the behaviour of BERT against the traditional TF-IDF vocabulary fed to machine learning algorithms. Our purpose of this work is to add empirical evidence to support or refuse the use of BERT as a default on NLP tasks. Experiments show the superiority of BERT and its independence of features of the NLP problem such as the language of the text adding empirical evidence to use BERT as a default technique to be used in NLP problems.
1 Introduction
The paper contrasts expert-designed linguistic methods and traditional TF-IDF-based machine learning with BERT, motivating empirical tests across text-classification scenarios and languages. It focuses on whether BERT can serve as a default NLP technique.
- Traditional NLP approaches use expert-selected linguistic features or statistical features such as bags of words and n-grams.Linguistic methods rely on rules and domain expertise, whereas machine-learning methods infer relevant features from annotated corpora.
- Multilingual traditional NLP systems require new rules when sentence structure or alphabets differ across languages.Resources such as UNL also require substantial expert effort and may not cover differing expression styles across contexts.
- BERT pretrains bidirectional representations from unlabeled text and then fine-tunes them with labeled data for different NLP tasks.The paper presents BERT as a model capable of producing state-of-the-art results across multiple tasks.
- The study compares BERT with traditional machine-learning classifiers trained on TF-IDF features across four text-classification experiments.The comparison is designed to test BERT against a representative traditional approach under different scenarios.
2 Related Work
Related work frames the paper around whether classical NLP remains useful relative to BERT, especially given concerns about data requirements and adversarial vulnerability. Prior empirical evidence favors BERT in low-shot settings, motivating tests across languages.
- The related-work question is whether classical NLP techniques remain useful compared with BERT’s reported state-of-the-art performance.The paper positions this comparison as a test of the continuing value of classical methods.
- BERT has been reported to outperform top classical machine-learning algorithms in low-shot classification with 100–1000 labeled examples per class.The cited comparison attributes this performance to deep transfer learning.
- A concern about BERT is that it requires huge amounts of text to deliver proper results.The paper tests this concern across different problems involving texts in different languages.
- Another critique is that BERT can be fooled by adversarial text, whereas symbolic approaches may solve some such errors.The paper proposes measuring BERT across languages to assess whether these concerns reflect common scenarios.
3 The BERT model and the traditional machine learning NLP methodology
The paper describes TF-IDF-based traditional classification and BERT’s Transformer-based pretraining and fine-tuning procedures. BERT processes token and sentence information through bidirectional self-attention and task-specific fine-tuning.
- 3.1 Term Frequency - Inverse Document Frequency (TF-IDF): TF-IDF builds a bag-of-words vocabulary by weighting terms according to term frequency and inverse document frequency.The paper defines tfidf(t_i) = tf_i · idf(t_i), with inverse document frequency based on document occurrence.
- 3.1 Term Frequency - Inverse Document Frequency (TF-IDF): The traditional experimental classifier uses TF-IDF features to construct a vocabulary for a machine-learning model.Further implementation details are deferred to the experiments section.
- 3.2 Bidirectional Encoder Representations from Transformers (BERT): BERT is pretrained on a large unlabeled corpus and then fine-tuned with labeled data for specific NLP tasks.Fine-tuning updates all parameters while task-specific inputs and outputs are plugged into the model.
- 3.2 Bidirectional Encoder Representations from Transformers (BERT): BERT uses a multi-layer bidirectional Transformer encoder with self-attention, feedforward sublayers, residual connections, and layer normalization.The encoder is described as a stack of six identical layers.
- 3.2 Bidirectional Encoder Representations from Transformers (BERT): Scaled dot-product attention maps queries, keys, and values into attention outputs, while multi-head attention performs this process through multiple learned projections.Self-attention uses queries, keys, and values originating from the same place.
- 3.2 Bidirectional Encoder Representations from Transformers (BERT): BERT represents sentences or sentence pairs with WordPiece tokens, special [CLS] and [SEP] tokens, and token, position, and segment embeddings.Segment embeddings distinguish tokens from the first and second sentences in a pair.
- 3.2 Bidirectional Encoder Representations from Transformers (BERT): BERT pretraining combines Masked Language Modeling, which predicts randomly masked tokens, with Next Sentence Prediction.For Next Sentence Prediction, half of sentence pairs are actual continuations and half are random pairings.
4 Experiments
Four text-classification experiments compare BERT with traditional TF-IDF-based machine-learning approaches across English, Portuguese, and Chinese settings. BERT outperforms the classical approach, including when language characteristics and dataset size vary.
- Experimental setup: The experiments use TF-IDF preprocessing with traditional classifiers and compare their results against a pretrained BERT model.Traditional implementations include sklearn models, AutoML, and directory-based BERT training.
- IMDB experiment: BERT outperforms the traditional models on the IMDB movie-review classification experiment.The paper reports that reaching comparable results with traditional approaches required substantially more work.
- RealOrNot tweets experiment: 0.83640 accuracy is reported for BERT on the RealOrNot competition, compared with 0.77607 for the best traditional classifier.The best traditional model was an H2O stacked ensemble.
- Portuguese news experiment: 0.91196 accuracy is reported for BERT on the Portuguese news competition, while GradientBoostingClassifier is identified as the best traditional classifier.The dataset contains nine news classes, and the experiment uses 75% of the data for training and 25% for validation.
- Chinese hotel reviews experiment: BERT outperforms the classical NLP approach in Chinese hotel-review sentiment classification despite the language using symbols that are not separated by spaces.The experiment tests positive and negative hotel reviews and uses 85% of the data for training and 15% for validation.
- Chinese hotel reviews experiment: A translated Chinese review meaning that the hotel’s view and service were very bad is correctly classified as negative by BERT.The prediction is made using Google Translator.
5 Conclusions and further work
Across four NLP scenarios, BERT outperformed the traditional TF-IDF-based approach and was less complicated to implement. The authors emphasize pre-training and transfer learning while acknowledging that BERT results can still be improved.
- BERT outperformed the traditional TF-IDF-based NLP approach across four NLP scenarios.
- BERT was less complicated to implement than the traditional methods.
- Pre-training enabled the reported results, with transfer learning especially apparent in the smallest-dataset experiment.
- The authors state that BERT appears suitable as a default for NLP tasks, but its results can be improved.
http://arxiv.org/ps/2005.13012v2
The supplied passages identify four figure files related to BERT and Transformer components, but provide no substantive descriptions of their contents.
- The paper references a figure file named BERT_encoder.jpg.
- The paper references a figure file named BERT_input.jpg.
- The paper references a figure file named BERT_prefine.PNG.
- The paper references a figure file named transformer_encoder.PNG.