PaperlayerView exact revision on arXiv

Source-linked AI summary

Improved Text Classification via Contrastive Adversarial Training

Lin Pan, Chung-Wei Hang, Avirup Sil, Saloni Potdar

arXiv:2107.10137v2cs.CL

TL;DR

The paper targets regularization for Transformer-based text-classification fine-tuning that encourages noise-invariant representations. CAT perturbs the embedding matrix and contrasts clean and adversarial examples, improving over standard fine-tuning across GLUE and intent-classification tasks.

Abstract

from arXiv · show

We propose a simple and general method to regularize the fine-tuning of Transformer-based encoders for text classification tasks. Specifically, during fine-tuning we generate adversarial examples by perturbing the word embeddings of the model and perform contrastive learning on clean and adversarial examples in order to teach the model to learn noise-invariant representations. By training on both clean and adversarial examples along with the additional contrastive objective, we observe consistent improvement over standard fine-tuning on clean examples. On several GLUE benchmark tasks, our fine-tuned BERT Large model outperforms BERT Large baseline by 1.7% on average, and our fine-tuned RoBERTa Large improves over RoBERTa Large baseline by 1.3%. We additionally validate our method in different domains using three intent classification datasets, where our fine-tuned RoBERTa Large outperforms RoBERTa Large baseline by 1-2% on average.

Introduction

The paper presents CAT, which regularizes Transformer-encoder fine-tuning for text classification with adversarial examples and a contrastive objective. It reports gains over clean-example fine-tuning baselines on GLUE and intent-classification tasks.

Introduction

  • Text's discrete nature prevents directly applying input perturbations, motivating adversarial regularization through word embeddings.Adversarial training uses small gradient-based perturbations and trains on both clean and perturbed examples; embedding perturbations do not map to new vocabulary words.Source: Introduction · p. 1
  • CAT applies adversarial training and a contrastive objective to regularize Transformer-based encoder fine-tuning for text classification.The paper builds on Miyato, Dai, and Goodfellow (2017), applying perturbations to the word embedding matrix and encouraging clean and paired perturbed examples to be close while separating unpaired examples.Source: Introduction · p. 1
  • CAT is evaluated on GLUE natural-language-understanding tasks and three dialog-system intent-classification datasets.The intent results include near-identical accuracy to the full-training-data baseline when using half the training data per intent.Source: Introduction · p. 1
  • BERTLarge improves by 1.7% on average and RoBERTaLarge by 1.3% on GLUE; intent RoBERTaLarge improves by 1% on full and 2% on difficult test sets.The baselines are standard fine-tuning on clean examples with cross-entropy loss.Source: Introduction · p. 1

Related Work

Prior work applies adversarial training to classification and adapts embedding perturbations to NLP, while contrastive-learning studies use varied ways to form positive pairs. CAT pairs each example with its adversarial counterpart; other approaches to improving Transformer classifiers include further pretraining and self-training.

Adversarial Training

  • Prior adversarial training spans object detection, segmentation, and image classification; this paper limits its discussion to white-box attacks.Text's discrete nature makes direct perturbation inapplicable, while Miyato, Dai, and Goodfellow (2017) adapt FGSM by perturbing word embeddings for NLP; CAT uses one-step FGSM with contrastive learning.Source: Adversarial Training · pp. 1–2

Contrastive Learning

  • Contrastive-learning work forms positive pairs through altered text views, nearby spans, same-class examples, dropout masks, or NLI pairs.CAT instead treats an original example and its adversarial example as a positive pair and uses contrastive loss as an additional fine-tuning regularizer.Source: Contrastive Learning · p. 2
  • CAT’s architecture combines FGSM perturbations to the encoder’s word embedding matrix with cross-entropy on clean and perturbed examples and a contrastive loss on paired representations.The contrastive loss brings the paired representations closer to promote noise-invariant representations.Source: Figure 1 · p. 2

Improving BERT for Text Classification

  • Other approaches to improving Transformer classifiers include target-domain language-model pretraining and self-training on unlabeled in-domain data.Self-training uses a teacher trained on labeled data to label data for a student model.Source: Improving BERT for Text Classification · p. 2
  • ELECTRA uses generated noisy text and token discrimination, while DeBERTa proposes disentangled attention and a form of VAT perturbing normalized word embeddings.Source: Improving BERT for Text Classification · p. 2

Method

CAT fine-tunes Transformer encoders with clean and adversarial examples, adding a contrastive objective that encourages clean and perturbed inputs to have similar sentence representations. The method combines standard classification with adversarial perturbation and contrastive learning.

Preliminaries

  • Standard fine-tuning adds a softmax classifier to the [CLS] sentence representation and trains it with cross-entropy loss.The setup assumes labeled examples and a pretrained Transformer encoder such as BERT or RoBERTa.Source: Preliminaries · p. 3

Adversarial Examples

  • CAT perturbs the Transformer word-embedding matrix using the loss gradient and trains on clean and adversarial examples; max-norm perturbations perform best among the tested forms.The authors also compare perturbing the embedding matrix with perturbing individual word embeddings, and report better results for the matrix perturbation.Source: Adversarial Examples · p. 3

Contrastive Learning

  • For contrastive learning, each clean [CLS] representation and its corresponding adversarial representation form a positive pair, while other batch examples serve as negatives.A nonlinear projection layer is applied, and the SimCLR formulation uses an InfoNCE objective.Source: Contrastive Learning · p. 4
  • CAT combines clean-example and adversarial-example classification losses with a weighted contrastive loss during fine-tuning.The contrastive objective encourages each clean example's sentence representation to resemble its corresponding adversarial example's representation.Source: Contrastive Learning · pp. 3–4

Experiment

Experiments evaluate CAT on GLUE and intent-classification datasets, including difficult-subset, sample-efficiency, and ablation tests. CAT improves over standard fine-tuning across the reported evaluations.

Datasets

  • Experiments cover seven GLUE tasks and CLINC, BANKING, and HWU; intent datasets have shorter texts and more classes, while difficult subsets select TF-IDF-dissimilar test examples.For CLINC, the experiments use in-scope examples only.Source: Datasets · pp. 4–5

GLUE Results

  • 1.7% average improvement over conventional fine-tuning for BERTLarge and 1.3% for RoBERTaLarge on GLUE; CAT also outperforms PGD and FreeAT and matches FreeLB overall.Some gains are not statistically significant: BERTLarge on MRPC and RTE, and RoBERTaLarge on MRPC, RTE, and CoLA.Source: Table 3 · p. 5, GLUE Results · p. 6

Intent Classification Results

Training Details

  • CAT uses baseline hyperparameters plus searches over epsilon, tau, and lambda; GLUE baselines train for three epochs and intent baselines for five.The intent-baseline settings follow Qi et al. (2021).Source: Training Details · p. 5

Sample Efficiency

  • With about half the training data per intent, CAT achieves near-identical accuracy to standard fine-tuning with all training data.The reduced training examples are randomly sampled from each dataset's full training data.Source: Table 6 · p. 6, Sample Efficiency · p. 7

Ablation

  • On GLUE, AT alone adds 0.8% over baseline and contrastive learning adds 0.5%; on difficult intent sets, the respective gains are 0.7% and 1.3%.Full CAT performs best on all GLUE tasks except CoLA. Matrix perturbation beats word-embedding perturbation by 0.6% for BERTLarge and 0.4% for RoBERTaLarge; max-norm selection beats random max/l2 selection by 0.4%.Source: Table 7 · p. 6, Table 8 · p. 6, Ablation · p. 7

Conclusion

CAT regularizes Transformer-encoder fine-tuning by combining adversarial training and contrastive learning. The paper reports consistent gains over standard fine-tuning across GLUE and intent-classification evaluations, with sample-efficiency and ablation tests supporting the combined objectives.

Conclusion

  • CAT consistently outperforms standard fine-tuning for text classification across GLUE tasks and three intent-classification datasets.The evaluations include different settings, and sample-efficiency and ablation tests show positive effects from combining the adversarial and contrastive objectives.Source: Conclusion · p. 7
  • The method regularizes fine-tuning of Transformer-based encoders by leveraging adversarial training and contrastive learning.Source: Conclusion · p. 7
  • The paper identifies complementary word-level objectives as future work to extend sentence-level contrastive learning to other NLP tasks.Source: Conclusion · p. 7

Related papers

Loading 2107.10137v2…