Source-linked AI summary
BERT: A Review of Applications in Natural Language Processing and Understanding
M. V. Koroteev
TL;DR
Text analysis needs models that represent polysemantic words according to context, while annotation evaluation lacks a generally accepted standard. This review explains BERT’s bidirectional masked-language-model approach and finds improved performance across subsequent natural-language applications.
Problem
Text classification requires distinguishing polysemantic words whose meanings vary with context, while text annotation increasingly exceeds the practical scope of manual methods.
Method
The review describes BERT’s masked-language-model training, which predicts randomly masked words from surrounding context using a deep bidirectional transformer.
Results
Across subsequent applications, BERT improved results and achieved 4.5% and 7% average superiority over known models on standard and large GLUE networks, respectively.
Takeaways & Limitations
BERT established pretrained representations on large datasets as a general basis for task-specific algorithms and demonstrated the advantage of bidirectional transformer-based comprehension.
Takeaways & Limitations
Text annotation model performance remains difficult to evaluate because no generally accepted annotation standard or automated quality metric exists, requiring manual assessment.
Abstract
from arXiv · showhide
In this review, we describe the application of one of the most popular deep learning-based language models - BERT. The paper describes the mechanism of operation of this model, the main areas of its application to the tasks of text analytics, comparisons with similar models in each task, as well as a description of some proprietary models. In preparing this review, the data of several dozen original scientific articles published over the past few years, which attracted the most attention in the scientific community, were systematized. This survey will be useful to all students and researchers who want to get acquainted with the latest advances in the field of natural language text analysis.
Basic concepts about the new BERT linguistic model
BERT is a bidirectional transformer language model that learns representations by predicting masked words from surrounding context. It is pretrained on unlabeled data and then adapted to labeled downstream tasks, achieving strong benchmark results.
- Learning objective: BERT trains a deep bidirectional transformer by predicting randomly selected masked words from their surrounding context.This masked-language-model objective addresses the limitation of unidirectional text scans.
- Training process: BERT uses two stages: pre-training on unlabeled data followed by additional training on labeled data for a specific application.Retraining procedures and architectures may differ by task while using the same model and parameter set.
- Architecture: The architecture is a multilayer bidirectional transformer with standard and large versions containing 110 million and 340 million trained parameters, respectively.The standard version has 12 layers and 768 coordinates, while the large version has 24 layers and 1024 coordinates.
- Benchmark results: On GLUE, BERT showed average superiority of 4.5% and 7% for standard and large neural networks, respectively, compared with the best-known models.GLUE evaluates natural language comprehension across multiple tasks and datasets.
- Benchmark results: On Adversarial Generations, BERT achieved an accuracy of 86.3, exceeding the human expert accuracy of 85.0.The dataset contains questions with four possible answers to choose from.
BERT Retraining Methodology for TextProblems Text
BERT can be adapted to text classification by task-specific retraining and one additional neural layer. Retraining improves performance, with intra-subject data generally more effective than intra-task data, while cross-subject training provides little benefit.
- Task Adaptation: BERT supports text classification through task-specific additional training with one added neural layer.The approach is reported to achieve the best current performance in text classification after retraining according to the specified methodology.
- Further Pre-training: Further pre-training can use intra-task, intra-subject, or cross-subject corpora, depending on the target texts and available data.This addresses differences between BERT’s general-purpose training distribution and the target application domain.
- Retraining Outcomes: Retrained models substantially outperform models without retraining, and intra-subject learning generally exceeds intra-task learning.Cross-subject learning does not significantly improve performance over the original BERT model.
- Retraining Strategies: Retraining strategies include using BERT representations as classification features or extracting information from its internal layers.Different layers can provide different syntactic and semantic information, with earlier layers containing more general information.
- Retraining Stability: Low learning rates can mitigate catastrophic forgetting during additional training on task-specific data.The cited study used a learning rate of about 1e-5.
The problem of improving the subject-specific classification of texts using BERT
Traditional token embeddings ignore context, making polysemous words difficult to distinguish in text classification, while subject-specific models require manual architecture search and intensive training. BERT is presented as a universal, pretrained text model intended to reduce these demands, although its full potential remains unrealized.
- Traditional limitations: Traditional embeddings assign each token a context-independent semantic vector calculated from a fixed surrounding window.This representation follows the assumption that similarly meaningful words occur in similar contexts.
- Traditional limitations: Fixed token representations hinder classification when polysemous words carry different meanings in different contexts.Distinguishing these context-dependent meanings may be especially important for indicative words.
- Traditional limitations: Traditional subject-specific models depend strongly on neural-network architectures designed for each domain, requiring manual architecture search and intensive training.This creates a separate model-design burden for each subject area.
- Contextual models: Contextual models such as CoVe and ELMo produce token representations that vary with the surrounding environment and can provide additional features for task-specific models.CoVe uses an attention-equipped LSTM encoder, while ELMo uses a bidirectional memory network trained for language modeling.
- Universal pretrained models: Universal pretrained models, including ULM-FiT, OpenAI GPT, and BERT, aim to support specific tasks with limited additional training and reduce the need for task-specific architectures.BERT is reported as more efficient than other text embeddings, but researchers identify further work on long sequences and small datasets as necessary.
Using BERT for Text Annotation Tasks
BERT-based text annotation improves perceived quality and enables variable-length representations, but evaluation remains manual because no accepted annotation standard or automated metrics exist. Separating indicative sentences in long texts remains difficult, motivating further research on datasets and quantitative evaluation metrics.
- Motivation: Manual annotation is difficult to scale as information volume increases, motivating automatic selection of key phrases and sentences that characterize a text.Manual annotation may remain viable for limited datasets, such as publicly available MIT lecture transcriptions.
- Representation flexibility: Variable-length annotations can be created by clustering vector representations of text with an arbitrary size K.The resulting dynamic presentation of the text is enabled by the vector representation produced through nesting.
- Evaluation: Text annotation model evaluation requires manual quality assessment because no generally accepted annotation standard or automated metrics exist.This lack of standardization complicates performance evaluation.
- Quality and limitations: BERT-based annotation improves the perceived quality of text annotation but struggles to separate indicative sentences that comprise a small proportion of the source text.Annotating long texts remains a promising research area, alongside developing model datasets and quantitative effectiveness metrics.
BERTScore: BERT Based
BERTScore evaluates generated text by comparing contextual BERT representations of reference and candidate sentences, addressing the limitations of surface-based metrics. It correlates more closely with human judgments than classic metrics but is slower than SacreBLEU.
- Motivation and contribution: Surface-based metrics such as BLEU emphasize superficial lexical and syntactic similarity, which can prefer less semantically faithful translations.The example contrasts “People like to travel abroad” with the semantically closer “Customers prefer foreign cars.”
- Method: BERTScore computes cosine distances between contextualized word representations in reference and candidate sentences, selecting closest candidate words and averaging their distances.It also derives precision-like, recall-like, and F1 scores, and can weight matches using smoothed reference-based IDF values.
- Motivation and contribution: BERTScore was proposed in February 2020 to automatically assess natural-language text-generation quality and support more effective model selection.Its authors argue that it correlates better with human quality judgments.
- Results: BERT-based similarity metrics show consistently higher results than classic text metrics and are statistically significantly closer to human estimates.The review situates BERTScore within metrics designed to correlate closely with human judgments.
- Limitations: About three times slower than SacreBLEU, BERTScore trades increased estimation accuracy for reduced evaluation speed.The review states that this additional validation cost should generally have little effect on machine-learning performance for typical text-processing datasets.
BERT Based Attacks on Text Classification
Text classification models can be fooled by adversarial examples that remain human-consistent, semantically similar, and linguistically correct. TEXTFOOLER constructs such examples in a black-box setting, reducing classification efficiency sharply, while adversarial retraining improves performance on adversarial data.
- Motivation: Adversarial examples can appear indistinguishable from real inputs to humans while causing machine-learning models to produce incorrect outputs.This vulnerability raises concerns about model safety and reliability.
- Requirements: Textual adversarial examples must preserve human interpretation, semantic meaning, and syntactic correctness.Existing approaches include misspelling, deleting, or removing words from text.
- TEXTFOOLER: TEXTFOOLER generates adversarial examples for black-box text-classification models, including BERT.Its procedure ranks word importance, transforms selected words, and uses synonym candidates filtered for grammatical and semantic consistency.
- Attack Results: 80-97% to 0-20%: generated adversarial examples reduce text-classification efficiency.The resulting performance change is presented as evidence of a successful attack on the machine-learning model.
- Adversarial Training: 2-7 percentage points: adversarial retraining raises classification efficiency on the test adversarial dataset compared with training without it.The improvement follows adding the generated adversarial dataset and training the model on it.
- Motivation: Studying model stability and automated attacks is relevant to improving the development and testing of intelligent systems.The paper presents robustness analysis as important for the broader deployment of such systems in decision-making.
Investigation of the robustness of BERT learning
The review reports that expanding BERT’s training data improved performance across GLUE, SQuAD, and RACE, indicating that the original model was undertrained. Further analyses produced RoBERTa, which improved the original model by an average of 3–4 percentage points across tested tasks.
- Training used approximately 160 GB of additional English-language corpora compared with BERT’s original 16 GB of BookCorpus and Wikipedia.The added corpora included CC-News, OpenWebText, and Stories.
- Performance improved on GLUE, SQuAD, and RACE, suggesting that the original BERT model was undertrained.The evaluation covered three benchmarks for natural language understanding.
- Removing the next sentence prediction function improved performance on subsequent tasks, while replacing segments with sentences degraded performance.The finding contradicted the original publication and was attributed to reduced long-term generalization.
- Increasing BERT’s training batch size produced benefits only to a limited extent.The study examined larger batches together with corresponding learning-rate increases.
- RoBERTa combined the best analyzed training practices and improved the original BERT model by an average of 3–4 percentage points.It reached the best reported values in all tested problems.
Conclusion
The conclusion presents BERT as a transformative model for intelligent natural language processing, emphasizing its widespread use, subsequent improvements, and bidirectional transformer-based contextual comprehension.
- Conclusion: BERT is now used in almost all word processing problems after provoking an intense reaction from the scientific community.Proposals for improving BERT appeared almost immediately and improved its results in subsequent problems.
- Conclusion: BERT demonstrated the advantage of bidirectional contextual text-comprehension models built on transformers with an attention mechanism.
- Conclusion: Further architectural improvements and training fine-tuning are expected to improve NLP algorithms for classification, annotation, machine translation, and question-answer systems.The passage anticipates additional scientific results from applying and adapting BERT to natural-language processing problems.