PaperlayerView exact revision on arXiv

Source-linked AI summary

DeBERTa: Decoding-enhanced BERT with Disentangled Attention

Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen

arXiv:2006.03654v6cs.CLcs.LG

TL;DR

Existing pre-trained language models do not fully capture how both content and position shape word relationships and masked-token predictions. DeBERTa addresses this with disentangled attention, an enhanced mask decoder, and virtual adversarial training, achieving consistently stronger downstream performance and surpassing human macro-average performance on SuperGLUE with its 1.5-billion-parameter model.

  • Problem

    Existing masked-language-modeling approaches may lack absolute-position information needed to distinguish words with similar local contexts.

  • Method

    DeBERTa separates content and position representations, uses disentangled attention and an enhanced absolute-position mask decoder, and applies virtual adversarial training during fine-tuning.

  • Results

    The 1.5-billion-parameter DeBERTa model surpasses human performance on SuperGLUE in macro-average score, while smaller models consistently outperform comparable pre-trained language models across downstream tasks.

  • Takeaways & Limitations

    DeBERTa substantially improves pre-training efficiency and downstream-task performance relative to earlier Transformer-based language models.

  • Takeaways & Limitations

    Despite its SuperGLUE results, DeBERTa does not reach human-level NLU intelligence and lacks compositional generalization across novel tasks.

Abstract

from arXiv · show

Recent progress in pre-trained neural language models has significantly improved the performance of many natural language processing (NLP) tasks. In this paper we propose a new model architecture DeBERTa (Decoding-enhanced BERT with disentangled attention) that improves the BERT and RoBERTa models using two novel techniques. The first is the disentangled attention mechanism, where each word is represented using two vectors that encode its content and position, respectively, and the attention weights among words are computed using disentangled matrices on their contents and relative positions, respectively. Second, an enhanced mask decoder is used to incorporate absolute positions in the decoding layer to predict the masked tokens in model pre-training. In addition, a new virtual adversarial training method is used for fine-tuning to improve models' generalization. We show that these techniques significantly improve the efficiency of model pre-training and the performance of both natural language understanding (NLU) and natural langauge generation (NLG) downstream tasks. Compared to RoBERTa-Large, a DeBERTa model trained on half of the training data performs consistently better on a wide range of NLP tasks, achieving improvements on MNLI by +0.9% (90.2% vs. 91.1%), on SQuAD v2.0 by +2.3% (88.4% vs. 90.7%) and RACE by +3.6% (83.2% vs. 86.8%). Notably, we scale up DeBERTa by training a larger version that consists of 48 Transform layers with 1.5 billion parameters. The significant performance boost makes the single DeBERTa model surpass the human performance on the SuperGLUE benchmark (Wang et al., 2019a) for the first time in terms of macro-average score (89.9 versus 89.8), and the ensemble DeBERTa model sits atop the SuperGLUE leaderboard as of January 6, 2021, out performing the human baseline by a decent margin (90.3 versus 89.8).

1 INTRODUCTION

DeBERTa is a Transformer-based language model that improves previous state-of-the-art pre-trained language models through disentangled attention and an enhanced mask decoder. Its techniques improve pre-training efficiency and downstream NLP performance, including consistent gains over RoBERTa-Large on several NLU tasks.

  • Background: Transformers apply self-attention in parallel, enabling greater large-scale training parallelization than recurrent neural networks (RNNs) (Vaswani et al., 2017).The introduction positions DeBERTa within the broader rise of large-scale pre-trained language models.
  • Model contributions: DeBERTa introduces disentangled attention, representing each word with separate content and position vectors and computing attention from content and relative-position information.This differs from BERT’s summed content and position embeddings.
  • Model contributions: The enhanced mask decoder incorporates context-word content and absolute-position information when predicting masked tokens during pre-training.Disentangled attention already captures context content and relative positions, but not absolute positions.
  • Training method: DeBERTa also proposes virtual adversarial training for fine-tuning, improving models’ generalization on downstream NLP tasks.The method is designed for fine-tuning pre-trained language models.
  • Empirical results: +3.6% on RACE (83.2% vs. 86.8%), +2.3% on SQuAD v2.0 (88.4% vs. 90.7%), and +0.9% on MNLI (90.2% vs. 91.1%) versus RoBERTa-Large.The DeBERTa model used half the training data and performed consistently better across a wide range of NLU tasks.

2 BACKGROUND

The section introduces Transformer language models and explains that DeBERTa’s disentangled attention separately represents word content and position. It also reviews masked language model pre-training, where models reconstruct randomly masked tokens.

  • Transformer Architecture: DeBERTa represents each input word with separate content and position vectors, addressing standard self-attention’s lack of a natural way to encode word position.Its disentangled attention computes word-to-word attention weights using separate matrices for content and relative position.
  • Transformer Architecture: Transformer language models consist of stacked blocks containing multi-head self-attention followed by a fully connected positional feed-forward network.
  • Masked Language Modeling: Masked language model pre-training corrupts text by randomly masking 15% of tokens and trains the model to reconstruct the masked tokens from the corrupted sequence.The objective is self-supervised and learns contextual word representations.
  • Masked Language Modeling: BERT keeps 10% of masked tokens unchanged, replaces another 10% with random tokens, and substitutes the remainder with the [MASK] token.

3 THE DEBERTA ARCHITECTURE

DeBERTa separates token content from relative position in attention and complements this with absolute-position information during masked-token decoding. Its implementation reuses relative-position embeddings to reduce memory requirements.

  • Disentangled Attention: DeBERTa computes attention from separate content and relative-position representations using content-to-content, content-to-position, position-to-content, and position-to-position terms.Each token is represented by content and position vectors, and word-pair attention is the sum of four disentangled attention scores.
  • Disentangled Attention: The relative-distance representation clips distances using a maximum relative distance k, with each distance mapped to an index in [0, 2k].For pre-training, the maximum relative distance is set to k = 512.
  • Efficient Implementation: Reusing shared relative-position projections across queries reduces the relative-position storage complexity from O(N^2d) to O(kd).The method indexes shared position embeddings with the relative-position matrix instead of allocating a separate embedding matrix for each query.
  • Enhanced Mask Decoder: DeBERTa adds absolute positions after the Transformer layers and before softmax, using them as complementary information when decoding masked tokens.Relative positions are modeled throughout the Transformer layers, while absolute positions help distinguish tokens with similar local contexts during MLM prediction.

4 SCALE INVARIANT FINE-TUNING

This section introduces Scale-invariant Fine-Tuning (SiFT), a virtual adversarial training algorithm for fine-tuning that improves stability by perturbing normalized word embeddings. It addresses instability caused by varying embedding norms, especially in larger models.

  • 4 SCALE INVARIANT FINE-TUNING: SiFT is a new virtual adversarial training variant for fine-tuning, based on algorithms described by Miyato et al. (2018) and Jiang et al. (2020).
  • 4 SCALE INVARIANT FINE-TUNING: Virtual adversarial training regularizes models by enforcing similar output distributions on task-specific examples and their adversarially perturbed versions.The perturbations are small and are intended to improve robustness to adversarial examples and model generalization.
  • 4 SCALE INVARIANT FINE-TUNING: In NLP, perturbations target word embeddings, whose norms vary across words and models; this variance increases in billion-parameter models and destabilizes adversarial training.
  • 4 SCALE INVARIANT FINE-TUNING: SiFT improves fine-tuning stability by applying adversarial perturbations to normalized word embeddings.When fine-tuning DeBERTa, the algorithm first normalizes word embeddings into stochastic vectors and then perturbs those normalized vectors; the authors find that this substantially improves performance.

5 EXPERIMENT

DeBERTa consistently outperforms comparable pre-trained language models across GLUE, other NLU benchmarks, and SuperGLUE, despite using less pre-training data in several comparisons. Ablations show that its components contribute materially, while scaling to 1.5B parameters enables performance beyond the SuperGLUE human baseline.

  • GLUE results: On MNLI, DeBERTa significantly outperforms all existing similar-size PLMs and establishes a new state of the art.The GLUE comparison covers eight NLU tasks, with DeBERTa pretrained on 78G of data versus 160G for RoBERTa, XLNet, and ELECTRA.
  • Large-model results: DeBERTa outperforms previous similar-size PLMs on all seven evaluated NLU tasks, including RACE by +1.4% (86.8% vs. 85.4%) over XLNet.Although Megatron1.3B is three times larger, DeBERTa outperforms it on three of four reported benchmarks.
  • Base-model results: Across three base-model tasks, DeBERTa exceeds RoBERTa and XLNet by larger margins than in large models, including +1.2% on MNLI-m over RoBERTa and +2% over XLNet.DeBERTabase scores 88.8% versus 87.6% for RoBERTabase and 86.8% for XLNetbase on MNLI-m.
  • Ablation study: Removing any DeBERTa component causes a performance drop, with -EMD losing 1.4% on RACE and 1.2% on SQuAD v2.0.The -EMD variant also loses 0.3% on SQuAD v1.1, 0.2% on MNLI-m, and 0.1% on MNLI-mm.
  • Scaling DeBERTa: 89.9 versus 89.8: single DeBERTa1.5B surpasses human performance on SuperGLUE, while DeBERTaEnsemble reaches 90.3 versus 89.8 for the human baseline.These results are reported as of December 29, 2020 for the single model and January 6, 2021 for the ensemble.

6 CONCLUSIONS

DeBERTa improves BERT and RoBERTa through disentangled attention and an enhanced mask decoder. Comprehensive experiments show improved pre-training efficiency and downstream performance, including a 1.5-billion-parameter model surpassing human SuperGLUE performance while remaining short of human-level NLU intelligence.

  • 6 CONCLUSIONS: DeBERTa improves BERT and RoBERTa with disentangled attention and an enhanced mask decoder.Disentangled attention represents each word with separate content and position vectors and computes attention using content and relative-position matrices.
  • 6 CONCLUSIONS: The 1.5-billion-parameter DeBERTa model surpasses human performance on SuperGLUE in terms of macro-average score.The authors describe this as the first such result and an important milestone toward general AI.
  • 6 CONCLUSIONS: A comprehensive empirical study finds that these techniques improve model pre-training efficiency and downstream-task performance.The conclusion attributes the gains to the proposed techniques collectively.
  • 6 CONCLUSIONS: Despite its SuperGLUE result, DeBERTa does not reach human-level NLU intelligence.The authors identify compositional generalization—generalizing to novel task compositions from familiar constituents—as a capability humans handle extremely well.

A APPENDIX · A.1 DATASET

The appendix summarizes NLP application benchmarks spanning natural language understanding, reading comprehension, commonsense inference, and named entity recognition. It describes the task coverage, data sources, and distinguishing characteristics of each benchmark.

  • A.1 DATASET: Table 6 summarizes the NLP application benchmarks used across the paper’s evaluations.The benchmarks cover multiple NLP task families and application settings.
  • A.1 DATASET: GLUE comprises nine natural language understanding tasks, including question answering, acceptability, sentiment analysis, similarity, paraphrase detection, and natural language inference.
  • A.1 DATASET: SuperGLUE extends GLUE with eight more difficult natural language understanding tasks, including question answering, inference, coreference resolution, and word sense disambiguation.
  • A.1 DATASET: RACE is a large-scale machine reading comprehension dataset collected from English examinations for middle school and high school students in China.
  • A.1 DATASET: SQuAD v1.1 and v2.0 are reading comprehension benchmarks built from approximately 500 Wikipedia articles, with crowdsourced questions and answers; v2.0 adds unanswerable questions.
  • A.1 DATASET: SWAG is a 113k-question adversarial dataset for grounded commonsense inference, combining natural language inference with physically grounded reasoning.
  • A.1 DATASET: CoNLL 2003 is an English dataset drawn from varied text sources and containing four types of named entity.

A.2 PRE-TRAINING DATASET

DeBERTa is pre-trained on a deduplicated 78GB corpus combining Wikipedia, BookCorpus, OPENWEBTEXT, and STORIES, with 5% reserved for validation.

  • A.2 PRE-TRAINING DATASET: 78GB of deduplicated text combines Wikipedia, BookCorpus (Zhu et al., 2015), OPENWEBTEXT, and STORIES (Trinh & Le, 2018).The sources contribute 12GB, 6GB, 38GB, and 31GB before deduplication, respectively.
  • A.2 PRE-TRAINING DATASET: 5% of the training data is sampled as a validation set to monitor pre-training.

A.3 IMPLEMENTATION DETAILS

The implementation follows established transformer practices, using dynamic batching, span masking, Adam with weight decay for pre-training, and selected fine-tuning choices. The code builds on Hugging Face Transformers, FairSeq, and Megatron.

  • A.3 IMPLEMENTATION DETAILS: Pre-training uses dynamic data batching, span masking with spans up to three tokens, and Adam with weight decay.Detailed pre-training hyperparameters are listed in Table 8.
  • A.3 IMPLEMENTATION DETAILS: RAdam produced better and more robust fine-tuning results on some tasks, including CoLA, RTE, and RACE.
  • A.3 IMPLEMENTATION DETAILS: The implementation is based on Hugging Face Transformers, FairSeq, and Megatron.

A.3.1 PRE-TRAINING EFFICIENCY

This section evaluates pre-training efficiency by tracking fine-tuned downstream performance across training checkpoints and comparing DeBERTa with RoBERTa on MNLI and SQuAD v2.0.

  • A.3.1 PRE-TRAINING EFFICIENCY: Pre-training efficiency is assessed by fine-tuning checkpoints sampled every 150K steps and measuring MNLI accuracy and SQuAD v2.0 F1.The evaluation uses RoBERTabase and DeBERTabase checkpoints on two representative downstream tasks.
  • A.3.1 PRE-TRAINING EFFICIENCY: The study reports separate hyper-parameter configurations for DeBERTa pre-training and downstream-task fine-tuning.These settings are documented in Tables 8 and 9.
  • A.3.1 PRE-TRAINING EFFICIENCY: Figure 1 presents pre-training performance curves for DeBERTa and counterpart models on the MNLI and SQuAD v2.0 development sets.The curves compare downstream performance as a function of the number of pre-training steps.

A.4 MAIN RESULTS ON GENERATION TASKS … A.11 FURTHER IMPROVE THE MODEL EFFICIENCY

Across generation, scaling, architecture analysis, and efficiency studies, DeBERTa improves language-model perplexity and downstream performance while supporting long-context attention and more parameter-efficient pre-training. The model’s disentangled attention and enhanced mask decoder also produce distinct attention patterns and add measurable computational overhead.

  • A.4 MAIN RESULTS ON GENERATION TASKS: DeBERTa enables autoregressive generation by applying a triangular self-attention mask, and DeBERTa-MT jointly pre-trains on MLM and ARLM using Wikitext-103.The evaluation compares DeBERTa with RoBERTa, GPT-2, and Transformer-XL, using 200k pre-training steps.
  • A.4 MAIN RESULTS ON GENERATION TASKS: Lower perplexities on both Wikitext-103 development and test sets, with joint MLM and ARLM training reducing perplexity further; DeBERTa outperforms DeBERTa-AP.The results support placing absolute position embeddings in the enhanced mask decoder rather than the input layer.
  • A.5 HANDLING LONG SEQUENCE INPUT: With k = 512 and 24 layers, DeBERTa-large can theoretically handle sequences up to 24,528 tokens through stacked relative-position attention.The paper identifies extremely long-sequence handling as future work despite this byproduct benefit.
  • A.6 PERFORMANCE IMPROVEMENTS OF DIFFERENT MODEL SCALES: Larger DeBERTa models perform better on GLUE, while SiFT consistently improves performance across model scales.The comparisons are reported on the GLUE development set, with additional modifications examined for DeBERTa1.5B and DeBERTa900M.
  • A.7 MODEL COMPLEXITY: Disentangled attention adds about 49M parameters to DeBERTa-large and 14M to DeBERTa-base, increasing parameter counts by 13% and 12%, respectively.The additional position-to-content and content-to-position attention scores increase computation by 30% versus BERT or RoBERTa and about 15% versus XLNet.
  • A.8 ADDITIONAL DETAILS OF ENHANCED MASK DECODER: The enhanced mask decoder receives hidden states plus decoding information such as absolute positions or prior decoder outputs, using stacked decoder layers before the language-model head.Each decoder layer’s output becomes the next layer’s decoding input.
  • A.9 ATTENTION PATTERNS: DeBERTa’s attention patterns differ from RoBERTa’s: EMD reduces the visible self-attention diagonal, while vertical strips appear in the examined patterns.The comparison includes DeBERTa variants without EMD, content-to-position attention, and position-to-content attention.
  • A.10 ACCOUNT FOR THE VARIANCE IN FINE-TUNING: Averaging five random-seed runs and testing significance shows DeBERTa-base significantly outperforms RoBERTa-base on the reported downstream-task examples.The examples concern MNLI-matched and SQuAD v1.1, with p-value < 0.05.
Loading 2006.03654v6…