Source-linked AI summary
Conditional BERT Contextual Augmentation
Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, Songlin Hu
TL;DR
The paper addresses text augmentation’s difficulty in generating diverse replacements without violating sentence labels. It fine-tunes BERT with a label-conditioned masked-language-model objective and uses the resulting model for contextual augmentation, reporting improvements across six classification tasks and additional applicability to style transfer.
Problem
Text augmentation is needed to reduce overfitting, but generic text transformations can produce meaningless sentences or violate semantic and label correctness.
Method
The method retrofits BERT into conditional BERT by adding a label-conditional masked-language-model task for contextual sentence augmentation.
Results
Experiments across six text-classification tasks show obvious improvement over existing text data augmentation approaches, with applicability to convolutional and recurrent classifiers.
Takeaways & Limitations
Conditional BERT contextual augmentation produces label-compatible replacements and is also demonstrated for style transfer.
Abstract
from arXiv · showhide
We propose a novel data augmentation method for labeled sentences called conditional BERT contextual augmentation. Data augmentation methods are often applied to prevent overfitting and improve generalization of deep neural network models. Recently proposed contextual augmentation augments labeled sentences by randomly replacing words with more varied substitutions predicted by language model. BERT demonstrates that a deep bidirectional language model is more powerful than either an unidirectional language model or the shallow concatenation of a forward and backward model. We retrofit BERT to conditional BERT by introducing a new conditional masked language model\footnote{The term "conditional masked language model" appeared once in original BERT paper, which indicates context-conditional, is equivalent to term "masked language model". In our paper, "conditional masked language model" indicates we apply extra label-conditional constraint to the "masked language model".} task. The well trained conditional BERT can be applied to enhance contextual augmentation. Experiments on six various different text classification tasks show that our method can be easily applied to both convolutional or recurrent neural networks classifier to obtain obvious improvement.
1 Introduction
The paper introduces conditional BERT contextual augmentation to generate diverse text replacements while preserving label compatibility, addressing overfitting and the uncontrollability of generic text transformations. Experiments report clear improvements across six text-classification tasks and classifier types.
- Limited training data makes deep neural networks prone to overfitting, motivating data augmentation to generate additional samples.
- Generic transformations such as mirroring or cropping can produce meaningless text and cannot ensure semantic invariance or label correctness.
- Conditional BERT contextual augmentation fine-tunes BERT with an added label-conditional masked-language-model constraint for label-compatible replacements.
- The method supports wider-ranging substitutions than ordinary contextual augmentation while addressing the risk that context-only predictions conflict with sentence labels.
- The approach can be applied to convolutional and recurrent neural-network classifiers, and conditional BERT is also explored for style transfer.
- Experimental results show that the approach outperforms existing text data augmentation approaches.
- The authors describe the work as the first attempt to alter BERT into conditional BERT or apply BERT to text-generation tasks.
2 Related Work
Related work spans language-model pre-training, generative and replacement-based text augmentation, and conditional replacement constraints. The paper distinguishes its approach by using a deep bidirectional language model for more general replacement augmentation.
- Language-model pre-training and fine-tuning have been used to improve downstream natural-language-processing tasks.
- The paper retrofits BERT with a conditional masked-language-model task to generate sentences compatible with given labels.
- Existing augmentation includes sampling, generative models, neighboring-word replacement, and WordNet synonym substitution, but generated quality or label compatibility can be difficult to guarantee.
- Unlike prior conditional replacement work, the paper uses a deep bidirectional language model whose attention mechanism provides structured memory for long-term dependencies.
3 Conditional BERT Contextual Augmentation
Conditional BERT replaces BERT’s segmentation embeddings with label embeddings and trains on a label-conditioned masked-language-model objective. It then generates augmented sentences by masking words and predicting replacements compatible with the original label.
- Bidirectional Language Model: A forward language model predicts a token sequence from left-to-right conditional probabilities, while a backward language model predicts it in the reverse direction.
- Bidirectional Language Model: Traditional bidirectional language modeling shallowly concatenates independently trained forward and backward language models.
- Masked Language Model Task: BERT’s masked-language-model task randomly masks input tokens and predicts only the masked words from their surrounding context.
- Conditional BERT: Conditional BERT retains BERT’s architecture but changes the input representation and training procedure.
- Conditional BERT: BERT’s segmentation embeddings are unrelated to annotated labels, so context-only predictions can produce replacements incompatible with sentence labels.
- Conditional BERT: The conditional masked-language-model objective predicts a masked token from both its context and label using p(·|y, S\{ti}) rather than p(·|S\{ti}).
- Conditional BERT: The model fine-tunes pre-trained BERT on labeled data, adapting segmentation embeddings to label embeddings and expanding their size for datasets with more than two labels.
- Contextual Augmentation: After training, the method masks words in labeled sentences, predicts label-compatible replacements, and adds the resulting sentences to the original dataset.
4 Experiment
Experiments evaluate conditional BERT contextual augmentation across six classification datasets, two classifier architectures, and several baselines. Conditional BERT performs best across the benchmark settings and can outperform BERT after only a few fine-tuning epochs.
- Datasets and classifiers: Experiments cover six benchmark classification datasets using CNN and LSTM-RNN sentence classifiers.The datasets include SST, Subj, MPQA, RT, and TREC variants; classifier hyperparameters are selected per task.
- Baselines: The comparison includes synonym replacement, contextual augmentation, label-conditional contextual augmentation, BERT, and conditional BERT.The reported table compares augmentation methods across two classifier architectures.
- Results: Conditional BERT contextual augmentation improves performance most across various datasets and classifier architectures.The authors report that BERT also helps, but less than conditional BERT, whose label conditioning supports compatibility with original labels.
5 Connection to Style Transfer
The paper connects conditional BERT contextual augmentation to style transfer because both modify words according to a target label or style while retaining sentence context. It applies the method by identifying style-relevant words and replacing them with conditional predictions.
- Connection: Conditional masked language modeling and style transfer both alter words according to a target property while preserving sentence context.The paper frames their connection through label-conditioned word changes without changing the surrounding context.
- Procedure: Style transfer uses attention-based word contributions to identify style-relevant tokens before conditional BERT predicts substitutes under the target style.The procedure first finds words relevant to the style, then masks them and predicts replacements using context and the target style label.
- Example: Reversing the sentiment label can transform a positive movie remark into a negative-style sentence, such as replacing “funny and interesting” with “boring and dull.”The example illustrates label-conditioned substitutions that produce a sentence with the opposite style.
6 Conclusions and Future Work
The paper concludes that conditional BERT improves sentence-classification augmentation over several baselines and can also support style transfer. Future work targets imbalanced datasets and paragraph- or document-level augmentation.
- Conclusions: The proposed method fine-tunes BERT with a conditional masked language-model task and applies the resulting model to sentence classification augmentation.Conditional BERT predicts replacements conditioned on both sentence context and labels.
- Conclusions: Experiments report that conditional BERT outperforms several baseline augmentation methods and can also be applied to style transfer.The conclusion summarizes results for sentence classification and the additional style-transfer application.