Source-linked AI summary
Adversarial Training for Large Neural Language Models
Xiaodong Liu, Hao Cheng, Pengcheng He, Weizhu Chen, Yu Wang, Hoifung Poon, Jianfeng Gao
TL;DR
The paper studies whether adversarial training can improve robustness without sacrificing generalization, especially during language-model pre-training. It proposes ALUM, which applies embedding-space virtual adversarial regularization across pre-training and fine-tuning, and reports gains across broad NLP evaluations, including for RoBERTa.
Problem
Adversarial training can improve robustness but often hurts generalization, while pretrained language models remain vulnerable to adversarial attacks.
Method
ALUM regularizes large language-model training by applying perturbations in embedding space that maximize adversarial loss during pre-training or fine-tuning.
Results
ALUM improves both generalization and robustness across a wide range of NLP tasks, including for extremely well-trained RoBERTa models through continual pre-training.
Takeaways & Limitations
Adversarial pre-training provides a promising direction for reconciling the apparent conflict between generalization and robustness, with additional gains from adversarial fine-tuning.
Takeaways & Limitations
Future work includes studying the roles of adversarial pre-training, accelerating adversarial training, and applying ALUM to other domains.
Abstract
from arXiv · showhide
Generalization and robustness are both key desiderata for designing machine learning methods. Adversarial training can enhance robustness, but past work often finds it hurts generalization. In natural language processing (NLP), pre-training large neural language models such as BERT have demonstrated impressive gain in generalization for a variety of tasks, with further improvement from adversarial fine-tuning. However, these models are still vulnerable to adversarial attacks. In this paper, we show that adversarial pre-training can improve both generalization and robustness. We propose a general algorithm ALUM (Adversarial training for large neural LangUage Models), which regularizes the training objective by applying perturbations in the embedding space that maximizes the adversarial loss. We present the first comprehensive study of adversarial training in all stages, including pre-training from scratch, continual pre-training on a well-trained model, and task-specific fine-tuning. ALUM obtains substantial gains over BERT on a wide range of NLP tasks, in both regular and adversarial scenarios. Even for models that have been well trained on extremely large text corpora, such as RoBERTa, ALUM can still produce significant gains from continual pre-training, whereas conventional non-adversarial methods can not. ALUM can be further combined with task-specific fine-tuning to attain additional gains. The ALUM code is publicly available at https://github.com/namisan/mt-dnn.
1 Introduction
The paper addresses the tension between generalization and robustness by introducing ALUM, a unified adversarial-training algorithm for large language-model pre-training and fine-tuning. Across broad NLP evaluations, ALUM improves both capabilities, including for extremely well-trained models.
- Motivation: Pretrained models such as BERT improve generalization but can remain highly vulnerable to adversarial attacks.On Yelp, BERT classification accuracy reportedly drops from 95.6% on standard testing to 6.8% on robust testing.
- Motivation: Prior adversarial-training work often hurts generalization and has largely focused on task-specific fine-tuning rather than pre-training.The paper identifies Wang et al. as a notable exception, limited to generative language modeling.
- Approach: ALUM applies adversarial training to large language-model pre-training and fine-tuning by maximizing adversarial loss through embedding-space perturbations.The method is designed as a general algorithm applicable on top of Transformer-based language models.
- Evaluation: ALUM is comprehensively evaluated across pre-training from scratch, continual pre-training, task-specific fine-tuning, and their combinations on generalization and robustness tasks.Benchmarks include GLUE, SQuAD, SNLI, SciTail, ANLI, HELLASWAG, SWAG, and Adversarial SQuAD.
- Results: ALUM substantially improves generalization and robustness over prior methods, including for extremely well-trained RoBERTa models.Continual pre-training without adversarial training fails to attain gains for RoBERTa, whereas ALUM does.
2 Preliminary
The preliminary section describes Transformer-based language-model inputs and BERT-style self-supervision. It focuses on subword tokenization, embeddings, masked language modeling, and next-sentence prediction.
- Input Representation: Inputs are text spans separated by [SEP], with words represented using fixed-vocabulary subword units produced by BPE or related methods.Subword tokenization addresses out-of-vocabulary words while compactly representing training text.
- Transformer Models: Transformer models combine token, position, and segment embeddings and use multi-head attention to model long-range dependencies.The section contrasts their parallel computation and dependency modeling with recurrent networks such as LSTM.
- Self-Supervision: Masked Language Modeling pre-trains BERT by replacing selected tokens and predicting the original tokens as a self-supervised cloze task.BERT and RoBERTa select 15% of tokens, using masking, unchanged tokens, and random replacements.
- Self-Supervision: The experiments gradually increase the masked-token rate from 5% to 25% during pre-training to improve training stability.The rate increases in 5% increments every 20% of training epochs.
- Self-Supervision: BERT also uses Next Sentence Prediction, a binary task that determines whether one sentence follows another in the original text.The paper retains NSP for fair comparison with BERT despite debate about its usefulness.
3 ALUM (Adversarial training for large neural LangUage Models)
ALUM augments standard language-model training with virtual adversarial regularization in embedding space, making the same framework applicable to pre-training and fine-tuning. The paper motivates this design as a way to improve robustness while preserving or improving generalization.
- Standard Objectives: Standard pre-training and fine-tuning minimize empirical training error using self-supervised or task-supervised objectives, respectively.Pre-training uses MLM and NSP, whereas fine-tuning uses labeled examples.
- Motivation: Large language models generalize well but can suffer severe adversarial degradation, while prior adversarial training often creates a generalization–robustness conflict.NLP attacks can involve replacing only a few words while preserving sentence semantics.
- Related Work: Prior NLP adversarial-training research rarely addressed pre-training, with the main exception applying it only to generative LSTM language modeling.The paper positions ALUM as addressing this gap for large Transformer-based language models.
- ALUM Algorithm: ALUM perturbs subword embeddings rather than discrete input text and regularizes the standard objective with virtual adversarial training.The adversarial term favors label smoothness in the embedding neighborhood.
- ALUM Algorithm: The smoothing coefficient α controls the trade-off between standard and robust errors, with α = 10 for pre-training and α = 1 for fine-tuning.The paper reports virtual adversarial training as especially useful when self-supervised labels may be noisy.
- Optimization: ALUM estimates perturbations using projected gradient steps, and the experiments set K = 1 to balance speed and performance.Larger K gives a better approximation but increases computational cost.
- Empirical Outcome: The paper reports that adversarial pre-training improves both generalization and robustness across a wide range of NLP tasks, including when applied to BERT and RoBERTa.It also reports further gains when adversarial pre-training is combined with adversarial fine-tuning.
4 Experiments
The experiments evaluate ALUM across adversarial pre-training from scratch, continual pre-training, and task-specific fine-tuning on broad generalization and robustness benchmarks. ALUM improves performance across standard and adversarial settings, including for well-trained RoBERTa models.
- Experimental setup: ALUM is evaluated through pre-training from scratch, continual pre-training, and task-specific fine-tuning across diverse NLP benchmarks.The benchmarks cover generalization, robustness, and tasks including classification, ranking, regression, named entity recognition, textual entailment, and machine reading comprehension.
- Improving Generalization: ALUMBERT-BASE gains 2.3% points in SQuAD v1.1 F1 over BERTBASE and 1.2% points over BERT+BASE.The comparison uses standard fine-tuning without adversarial training, while BERTBASE and ALUMBERT-BASE use 1M pre-training steps and BERT+BASE uses 1.6M.
- Improving Generalization: ALUM outperforms standard BERT on all three biomedical NER datasets despite the domain differing substantially from the pre-training corpus.The datasets are BC2GM, NCBI, and JNLPBA, evaluated with entity-level F1 scores.
- Improving Generalization: ALUM improves continual pre-training for RoBERTa, whereas standard continual pre-training fails to improve downstream generalization.ALUMROBERTA-BASE gains +0.5% on MNLI development data over RoBERTaBASE, and ALUMROBERTA-LARGE gains +0.7% over RoBERTaLARGE.
- Improving Robustness: On ANLI, ALUMROBERTA-LARGE gains 7.3% points in test accuracy over RoBERTaLARGE and 5.0% points over XLNet.ALUM also improves Adversarial SQuAD, including +6.4% and +5.0% F1 over BERTBASE in AddSent and AddOneSent settings.
- Combining Adversarial Pre-Training and Fine-tuning: Combining adversarial pre-training and fine-tuning produces the best results on representative development sets and yields similar gains across several additional tasks.On ANLI, ALUMRoBERTa-SMART exceeds ALUMROBERTA-LARGE by +1.1% accuracy and RoBERTaLARGE by +5.1% points.
5 Conclusion
The paper proposes ALUM and reports that adversarial pretraining improves both generalization and robustness for large neural language models. It also identifies future work in improving training speed and extending ALUM to other domains.
- ALUM is a general adversarial training algorithm for large neural language models.It supports adversarial training during pretraining and fine-tuning.
- Adversarial pretraining significantly improves both generalization and robustness across a wide range of NLP tasks.The reported gains apply to both BERT and RoBERTa.
- ALUM can be combined with adversarial fine-tuning to obtain further gains.
- Future work includes studying adversarial pretraining, speeding up adversarial training, and applying ALUM to other domains.
A NLP Application Benchmarks
The NLP application benchmarks cover diverse tasks and are used to assess generalization, robustness, and combined adversarial pretraining with fine-tuning. They include language understanding, entailment, named entity recognition, and machine reading comprehension datasets.
- GLUE is a nine-task NLU benchmark spanning question answering, acceptability, sentiment, similarity, paraphrase detection, and natural language inference.Its task diversity makes it suitable for evaluating generalization and robustness of NLU models.
- SNLI contains 570k human-annotated sentence pairs for natural language inference, while SciTail tests whether a premise entails a hypothesis.
- SQuAD v1.1 and v2.0 are machine reading comprehension benchmarks built from Wikipedia passages and crowdsourced questions and answers.SQuAD v2.0 additionally includes unanswerable questions.
- BC2GM, NCBI, and JNLPBA provide biomedical named entity recognition datasets covering genes, diseases, and molecular-biology taxonomies.
- Table 9 summarizes information about the NLP application benchmarks.