CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understanding

1 Introduction

• Adversarial examples preserve semantics, whereas contrastive examples change them; robustness research has largely focused on the former.Small changes in text can substantially alter sentence semantics.Source: §1 · pp. 1–2
• Adversarially trained models improve against adversarial attacks but perform worse on contrastive examples.The pilot experiments report a drop in contrastive-example performance alongside improved adversarial robustness.Source: §1 · p. 2
• CLINE generates adversarial and contrastive examples by unsupervised replacement of representative tokens, then learns contrastively from both.It uses external semantic knowledge, including WordNet, to generate examples.Source: §1 · p. 2
• +1.6% absolute improvement on 4 contrastive test sets and +0.5% on 4 adversarial test sets versus RoBERTa.The reported results indicate simultaneous gains in sensitivity to semantic changes and robustness to adversarial attacks.Source: §1 · p. 2

2.1 Model and Datasets

• The pilot uses TextFooler adversarial examples and contrastive sets to compare vanilla BERT and RoBERTa with FreeLB versions.The selected datasets include IMDB sentiment analysis and SNLI sentence-pair inference.Source: §2.1 · p. 2

2.2 Result Analysis

• FreeLB achieves higher accuracy than vanilla models on adversarial sets but drops on contrastive sets, especially for BERT.The authors conclude adversarial training is unsuitable for contrastive sets and can have negative effects.Source: §2.2 · p. 2, Table 2 · p. 3
• Adversarial training aims to keep labels unchanged, whereas contrastive examples make small, label-changing modifications.The authors describe the two example types as a natural contradiction that calls for additional strategies to detect fine-grained semantic changes.Source: §2.2 · pp. 2–3

2.3 Case Study

• In an IMDB case, vanilla BERT correctly predicts a negative review, while FreeLB-trained BERT treats negative-sentiment features as noise and predicts positive.The authors use this case to argue that adversarial training may not suit examples with changed semantics.Source: §2.3 · p. 3

3.1 Generation of Examples

• CLINE creates a synonym-based sentence close to the input and an antonym- or random-word-based sentence far from it, replacing about 40% and 20% of tokens, respectively.The method uses spaCy for segmentation and part-of-speech tagging, selecting verbs, nouns, adjectives, and adverbs for replacement.Source: §3.1 · p. 4

3.2 Training Objectives

• The training objectives combine masked language modeling with replaced-token detection on the generated sentences.The detection classifier predicts which tokens were replaced in the similar and different examples.Source: §3.2 · p. 4
• CLINE treats the original and synonym-based sentences as a positive pair and the original and antonym-based sentences as a negative pair, using [CLS] embeddings for sentence similarity.The contrastive objective uses one generated antonym-based sentence as its negative example, rather than randomly sampling negatives from the pre-training corpus.Source: §3.2 · p. 5
• The method combines masked language modeling, replaced-token detection, and contrastive learning with learned task weights.The combined loss is a weighted sum of the three objectives.Source: §3.2 · p. 5

4.2 Datasets

• Evaluation spans sentiment, natural-language inference, perspective classification, Boolean-answer reading comprehension, and four-topic news classification.Source: §4.2 · p. 5

4.3 Experiments on Contrastive Sets

• CLINE significantly outperforms other methods on contrast consistency and also improves performance on the original test set.Contrast consistency requires correct predictions on both the original and contrastive examples.Source: Table 4 · p. 6, §4.3 · p. 6

4.4 Experiments on Adversarial Sets

• CLINE reaches state-of-the-art performance against FreeLB on adversarial sets and exceeds vanilla BERT and RoBERTa accuracy on all four benchmarks.The authors also report that CLINE remains sensitive to semantic changes while robust to adversarial examples.Source: Table 5 · p. 6, §4.4 · p. 6

4.5 Ablation Study

• Removing hard negatives from CLINE causes significant performance degradation in most settings, whereas removing RTD causes only a little bit of degradation; the method shows prominent improvement with 1% and 10% supervised training data.Source: §4.5 · p. 7

4.6 Sentence Semantic Representation

• CLINE-B and CLINE-R improve sentence-level semantic-change Hits; the authors speculate that NSP training and insufficient training may explain differences between models.For BERT, [CLS] performs worse than mean-pooling; the authors say RoBERTa's [CLS] result has no meaning because its pretraining omits NSP.Source: Table 7 · p. 7, §4.6 · pp. 7–8

5.1 Pre-trained Language Models

• PLM pre-training largely focuses on token-level tasks, leaving global sentence semantics less modeled.Autoregressive pre-training predicts the next word, while denoising autoencoding reconstructs input sequences using left and right context.Source: §5.1 · p. 8

5.2 Adversarial Training

• Text adversarial training is challenging because discrete inputs are hard to optimize, despite many proposed defense methods.Prior approaches include word noise, paraphrases, and embedding perturbations; pilot experiments suggest robustness remains insufficient.Source: §5.2 · p. 8

5.3 Contrastive Learning

• Prior contrastive-learning work often uses simple random negatives, whereas this paper proposes negative examples with opposite semantics.Contrastive learning draws representations of positive pairs together and pushes negative pairs apart.Source: §5.3 · p. 8

• CLINE targets PLMs that are robust against adversarial attacks and sensitive to small semantic changes.Source: §6 · p. 8
• CLINE automatically generates adversarial and semantic negative examples from an original sentence and trains the model with three objectives.Source: §6 · p. 8
• Empirical results indicate that CLINE considerably improves PLMs' sensitivity while also gaining robustness.Source: §6 · p. 8