Source-linked AI summary
CERT: Contrastive Self-supervised Learning for Language Understanding
Hongchao Fang, Sicheng Wang, Meng Zhou, Jiayuan Ding, Pengtao Xie
TL;DR
Existing language-model pretraining tasks are mostly token-level and may not capture sentence-level semantics sufficiently. CERT continues pretraining a language encoder with sentence-level contrastive learning over back-translated augmentations, then evaluates it on GLUE, where it outperforms BERT on most tasks and in average score.
Problem
Existing pretraining approaches mostly define auxiliary predictive tasks on tokens, which may insufficiently capture sentence-level semantics.
Method
CERT uses back-translation to create sentence augmentations and continues pretraining a pretrained language encoder by predicting whether augmented pairs come from the same original sentence.
Results
CERT outperforms BERT on 7 of 11 GLUE tasks, matches it on 2, performs worse on 2, and achieves a better average score of 80.7.
Takeaways & Limitations
CERT demonstrates that sentence-level contrastive self-supervised learning can improve language representations over BERT across the GLUE benchmark overall.
Takeaways & Limitations
CERT uses about 400 KB of texts and a single GPU for a dozen hours, whereas comparable models use tens or hundreds of gigabytes and thousands of GPU machines for several days.
Abstract
from arXiv · showhide
Pretrained language models such as BERT, GPT have shown great effectiveness in language understanding. The auxiliary predictive tasks in existing pretraining approaches are mostly defined on tokens, thus may not be able to capture sentence-level semantics very well. To address this issue, we propose CERT: Contrastive self-supervised Encoder Representations from Transformers, which pretrains language representation models using contrastive self-supervised learning at the sentence level. CERT creates augmentations of original sentences using back-translation. Then it finetunes a pretrained language encoder (e.g., BERT) by predicting whether two augmented sentences originate from the same sentence. CERT is simple to use and can be flexibly plugged into any pretraining-finetuning NLP pipeline. We evaluate CERT on 11 natural language understanding tasks in the GLUE benchmark where CERT outperforms BERT on 7 tasks, achieves the same performance as BERT on 2 tasks, and performs worse than BERT on 2 tasks. On the averaged score of the 11 tasks, CERT outperforms BERT. The data and code are available at https://github.com/UCSD-AI4H/CERT
1. Introduction
Existing language-model pretraining tasks mostly predict tokens, which may insufficiently capture sentence-level semantics. CERT addresses this gap with sentence-level contrastive self-supervised learning using back-translated augmentations.
- Token-level predictive targets in existing pretraining approaches may not sufficiently capture global sentence-level semantics.
- CERT learns sentence-level representations by predicting whether two augmented sentences originate from the same original sentence.The method uses contrastive self-supervised learning rather than token-defined prediction tasks.
- Back-translation creates the sentence augmentations used for CERT's contrastive learning objective.
- CERT was evaluated on 11 GLUE natural language understanding tasks, with ablations examining augmentation methods and pretraining-corpus sources.
2. Pretraining of Transformers for Language Understanding
Transformer-based language representation models use self-attention to process text efficiently and capture dependencies between tokens. BERT applies this architecture bidirectionally and pretrains by predicting randomly masked tokens.
- Transformer replaces recurrent computation with self-attention, enabling parallel computation while capturing dependencies between tokens.Its building blocks combine self-attention with position-wise feed-forward layers.
- BERT is a multi-layer bidirectional Transformer encoder for representing texts.
- BERT pretraining randomly masks input tokens and predicts the masked tokens from their final hidden vectors.
- For downstream tasks, a task-specific layer can be added to pretrained BERT and trained with labeled target-task data.
3. Contrastive Self-supervised Learning
Contrastive self-supervised learning constructs augmented examples and trains representations to distinguish pairs from the same original example from pairs from different examples. MoCo implements this objective with a queue to avoid requiring very large minibatches.
- Contrastive self-supervised learning labels augmented pairs as similar when they share an original example and dissimilar otherwise.
- In the queue-based contrastive setup, a query is compared with keys, and similarity scores plus binary pair labels define the contrastive losses.
- SimCLR uses feature extraction and projection modules to learn representations from similar and dissimilar image pairs.Its downstream feature extractor is retained after training while the projection module is discarded.
- MoCo stores augmented examples in a dynamic queue whose size is independent of minibatch size.The queue adds the latest minibatch and removes the oldest minibatch each iteration.
4. CERT
CERT continues pretraining a pretrained language encoder with contrastive self-supervised learning on unlabeled target-task text, then fine-tunes it with target-task labels. Back-translation supplies sentence augmentations, while MoCo organizes positive and negative pairs through a momentum-updated queue.
- CERT workflow: CERT continues training a pretrained language model with contrastive self-supervised learning on unlabeled target-task input texts.The resulting pretrained CERT model is subsequently fine-tuned using target-task inputs and labels.
- Data Augmentation: Back-translation translates each target-task sentence into another language and back, treating the returned sentence as an augmentation.Multiple target languages can produce different augmentations of one source sentence.
- CSSL Pretraining: CSSL labels augmented sentences from the same original sentence as positive pairs and those from different sentences as negative pairs.
- CSSL Pretraining: CERT uses a queue of augmented-sentence keys and momentum updates for the key encoder during contrastive pretraining.The BERT encoder is fine-tuned by minimizing the resulting contrastive losses.
- CSSL Pretraining: Only target-task input sentences, not their labels, are used during CSSL pretraining; labels are introduced during downstream fine-tuning.
5. Experiments
CERT is evaluated against BERT across GLUE’s 11 natural language understanding tasks, including sentence, similarity, paraphrase, and inference settings. Results favor CERT overall, while ablations examine augmentation methods and CSSL pretraining corpora.
- Tasks and Datasets: GLUE contains 11 tasks spanning single-sentence, similarity and paraphrase, and inference evaluation settings.Accuracy is the default metric unless otherwise noted, with validation and test results reported separately.
- Results: On validation medians, CERT outperforms BERT on 5 tasks and is on par with BERT on the other 5 reported tasks.The median comparison covers CoLA, RTE, QNLI, SST-2, and QQP as CERT wins.
- Results: On GLUE tests, CERT outperforms BERT on 7 tasks, matches it on 2, and performs worse on 2, achieving an average score of 80.7.CERT wins on RTE, QNLI, STS-B, MRPC, MNLI-m, MNLI-mm, and QQP; it loses on CoLA and SST-2.
- Ablation on Data Augmentation: Back-translation generally outperforms EDA for CERT augmentation, with EDA using synonym replacement, insertion, swapping, or deletion.Back-translation performs global sentence-level augmentation, whereas EDA performs local word- or phrase-level edits.
- Ablation on CSSL Pretraining Corpora: CSSL corpus effects vary by task: CoLA benefits more from its task training data, while STS-B benefits more from the BERT pretraining corpus.The study therefore reaches no clear conclusion about which corpus is universally preferable.
6. Related Works
Prior language pretraining methods model token-level objectives, while contrastive self-supervised learning has been explored for representation learning beyond language.
- Language representation pretraining: GPT models next tokens with a Transformer decoder and maximize likelihood over token sequences.GPT-2 modifies GPT’s normalization configuration and uses byte pair encoding for input representation.
- Language representation pretraining: BART jointly trains an encoder and decoder by reconstructing randomly corrupted input text.Its corruption process includes token masking and token deletion.
- Contrastive self-supervised learning: Contrastive self-supervised learning has been studied for image recognition, reinforcement learning, and supervised representation learning.The cited work includes contrastive predictive coding for data-efficient image recognition and contrastive representations for reinforcement learning.
7. Conclusions and Future Works
CERT continues training pretrained language representation models with contrastive self-supervised learning on target-task inputs, then evaluates them across GLUE tasks. The authors report that CERT outperforms BERT on most evaluated tasks and propose ranking-based self-supervised losses for future work.
- Conclusions and Future Works: CERT continues training a pretrained language representation model on target-task inputs using contrastive self-supervised learning, then finetunes it with task labels.Back-translation generates augmented sentences, and the contrastive objective predicts whether two augmentations came from the same original sentence.
- Conclusions and Future Works: CERT outperforms BERT on the majority of 11 GLUE natural language understanding tasks on both test and validation sets.The authors present this as evidence of contrastive self-supervised learning’s effectiveness for language representations.
- Conclusions and Future Works: Future work will investigate ranking-based losses that order augmented sentences by their discrepancy from the original.The proposed auxiliary task predicts the order of a ranked list of augmentations.