PaperlayerView exact revision on arXiv

Source-linked AI summary

DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing

Pengcheng He, Jianfeng Gao, Weizhu Chen

arXiv:2111.09543v4cs.CLcs.LG

TL;DR

The paper addresses inefficient and unstable pre-training when DeBERTa is combined with ELECTRA-style objectives and shared token embeddings. It replaces MLM with RTD and introduces gradient-disentangled embedding sharing, yielding state-of-the-art results across English and multilingual NLU benchmarks. A stated scope boundary is that multilingual pre-training used no parallel data, which the authors expect could improve the model.

  • Problem

    Combining DeBERTa with ELECTRA-style training can create pre-training instability and inefficiency because generator and discriminator objectives interfere through shared embeddings.

  • Method

    DeBERTaV3 combines RTD and disentangled attention with gradient-disentangled embedding sharing, allowing discriminator access to generator semantics without interfering with generator gradients.

  • Results

    DeBERTaV3 achieves significant improvements across NLU tasks, including more than 1.37% over similarly structured models on GLUE and 3.6% over XLM-Rbase on XNLI cross-lingual transfer accuracy.

  • Takeaways & Limitations

    The results establish DeBERTaV3 as a state-of-the-art pre-trained language model across Large, Base, Small, and XSmall scales.

  • Takeaways & Limitations

    The multilingual model was trained without parallel data, and the authors expect continual pre-training on parallel data to produce a better model.

Abstract

from arXiv · show

This paper presents a new pre-trained language model, DeBERTaV3, which improves the original DeBERTa model by replacing mask language modeling (MLM) with replaced token detection (RTD), a more sample-efficient pre-training task. Our analysis shows that vanilla embedding sharing in ELECTRA hurts training efficiency and model performance. This is because the training losses of the discriminator and the generator pull token embeddings in different directions, creating the "tug-of-war" dynamics. We thus propose a new gradient-disentangled embedding sharing method that avoids the tug-of-war dynamics, improving both training efficiency and the quality of the pre-trained model. We have pre-trained DeBERTaV3 using the same settings as DeBERTa to demonstrate its exceptional performance on a wide range of downstream natural language understanding (NLU) tasks. Taking the GLUE benchmark with eight tasks as an example, the DeBERTaV3 Large model achieves a 91.37% average score, which is 1.37% over DeBERTa and 1.91% over ELECTRA, setting a new state-of-the-art (SOTA) among the models with a similar structure. Furthermore, we have pre-trained a multi-lingual model mDeBERTa and observed a larger improvement over strong baselines compared to English models. For example, the mDeBERTa Base achieves a 79.8% zero-shot cross-lingual accuracy on XNLI and a 3.6% improvement over XLM-R Base, creating a new SOTA on this benchmark. We have made our pre-trained models and inference code publicly available at https://github.com/microsoft/DeBERTa.

1 INTRODUCTION

The paper targets more efficient pre-training by combining DeBERTa with ELECTRA-style replaced token detection and by resolving harmful conflicts in shared embeddings. DeBERTaV3 variants achieve strong downstream NLU results, including new state-of-the-art performance among similarly structured models.

  • Energy-efficient PLM development seeks fewer parameters and lower computation cost while retaining high model capacity.
  • RTD replaces MLM by using a generator to create token corruptions and a discriminator to distinguish replaced tokens from originals.
  • Vanilla generator-discriminator embedding sharing hurts efficiency and performance because their different objectives pull token embeddings in opposite directions.
  • DeBERTaV3large surpasses similarly structured models on GLUE by more than 1.37% in average score.
  • DeBERTaV3base improves over DeBERTabase by 1.8% on MNLI-matched accuracy and 2.2% on SQuAD v2.0 F1.

2 BACKGROUND

The background contrasts conventional masked language modeling with DeBERTa’s disentangled attention and ELECTRA’s generator-discriminator replaced-token detection framework. These approaches differ in how they encode positions, corrupt inputs, and train token-level predictions.

  • Transformer language models stack self-attention blocks, but standard self-attention lacks a natural mechanism for encoding word position.
  • Positional information can be added as a bias to word embeddings, making representations depend on both content and position.
  • DeBERTa separates content and position vectors and computes attention using disentangled content and relative-position representations.
  • MLM randomly masks 15% of sequence tokens and trains a language model to reconstruct the masked tokens from the corrupted input.
  • ELECTRA uses a generator trained with MLM to produce ambiguous replacements and a discriminator trained with token-level binary classification to detect them.
  • ELECTRA jointly optimizes generator and discriminator losses, weighting the discriminator loss by λ = 50.

3 DEBERTAV3

DeBERTaV3 replaces DeBERTa’s MLM objective with RTD and introduces GDES to share embeddings while preventing conflicting discriminator gradients from updating the generator. Experiments report faster convergence and improved downstream NLU performance for GDES and DeBERTaV3.

  • 3 DEBERTAV3: DeBERTaV3 replaces DeBERTa’s MLM objective with RTD, combining DeBERTa’s disentangled attention with ELECTRA’s sample-efficient training objective.
  • 3.1 DEBERTA WITH RTD: +2.5% MNLI-m accuracy and +3.8% SQuAD v2.0 F1 are reported for DeBERTaV3 over DeBERTa under the evaluated pre-training setup.The implementation uses Wikipedia and BookCorpus, a batch size of 2048, and 125,000 training steps.
  • 3.2 TOKEN EMBEDDING SHARING IN ELECTRA: Vanilla embedding sharing jointly updates generator and discriminator embeddings, causing conflicting MLM and RTD gradients that slow convergence.The shared embeddings receive gradients from both tasks, producing a tug-of-war when their optimal directions differ.
  • 3.2 TOKEN EMBEDDING SHARING IN ELECTRA: NES separates generator and discriminator embeddings and updates them alternately, avoiding conflicting gradients and converging faster than ES.NES does not significantly improve MNLI or SQuAD v2.0 despite producing a more semantically coherent generator embedding matrix.
  • 3.3 GRADIENT-DISENTANGLED EMBEDDING SHARING: GDES shares embeddings while using a stop-gradient mechanism so RTD updates discriminator residual embeddings without affecting generator embeddings.The generator embeddings are updated with MLM, while discriminator residual embeddings receive RTD updates.
  • 3.3 GRADIENT-DISENTANGLED EMBEDDING SHARING: GDES matches NES’s convergence efficiency, preserves more discriminator semantic information than NES, and achieves the best MNLI and SQuAD v2.0 fine-tuning performance.The computation cost per training step is the same for ES, NES, and GDES because they differ only in embedding sharing.

4 EXPERIMENT

The experiments combine RTD, GDES, and disentangled attention in DeBERTaV3 models, evaluating them across English and multilingual NLU benchmarks. DeBERTaV3 consistently improves over comparable baselines, including larger gains for base and smaller models and strong multilingual transfer results.

  • Model Setup: DeBERTaV3 combines RTD, GDES, and disentangled attention in large, base, and small models using standard DeBERTa-based pre-training settings.The discriminator components of the large and base variants match those of the corresponding DeBERTa models.
  • Large Models: DeBERTaV3large establishes new results among similar-structure models on GLUE and is compared across eight NLU tasks.The GLUE evaluation uses a classification head over the final-layer [CLS] representation.
  • Large Models: DeBERTaV3large outperforms similar-size models on all six reported tasks, with gains of +2.4% on RACE and +2.6% on SWAG.The authors associate these gains with tasks requiring reasoning capability and common-sense knowledge, while presenting that interpretation as a conjecture.
  • Base and Smaller Models: DeBERTaV3base improves over both DeBERTabase and ELECTRAbase on MNLI-m, while its SQuAD v2.0 EM score exceeds ELECTRAbase by +4.9% and DeBERTabase by +2.3%.On MNLI-m, the reported score is 90.6% versus 88.8% for both baselines.
  • Base and Smaller Models: DeBERTaV3small improves over BERTsmall by 6.4% on MNLI-m and 9.7% F1 on SQuAD v2.0; DeBERTaV3xsmall performs on par or better despite half the parameters.The smaller variants also outperform corresponding MiniLMv2 models and, in some comparisons, larger RoBERTabase and XLNetbase models.
  • Multilingual Model: The multilingual model is evaluated on XNLI across 15 languages under zero-shot transfer and translate-train-all settings, with future parallel-data pre-training anticipated.The multilingual model uses CC100 data and a SentencePiece vocabulary, while the cited limitation concerns the absence of parallel-data training.
  • Overall Findings: Across a broad range of downstream tasks, the authors report consistent improvements and characterize DeBERTaV3 as efficient.The conclusion links the broad task improvements to the value of improving pre-trained language models.

5 CONCLUSIONS

The paper combines DeBERTa’s relative-position modeling with ELECTRA’s RTD objective, then introduces GDES to address generator–discriminator interference. DeBERTaV3 with GDES improves over prior models across NLU tasks and model scales, including English and cross-lingual benchmarks.

  • 5 CONCLUSIONS: The proposed paradigm combines DeBERTa and ELECTRA, using relative position encoding and replaced token detection.The paper presents this combination as its pre-training framework.
  • 5 CONCLUSIONS: GDES addresses RTD pre-training instability and inefficiency by letting the discriminator use generator embedding semantics without interfering with generator gradients.The paper identifies this generator–discriminator interference as the “tug-of-war” dynamics.
  • 5 CONCLUSIONS: DeBERTaV3Large exceeds similar-architecture models by more than 1.37% on GLUE average score, while mDeBERTaV3base beats XLM-Rbase by 3.6% on XNLI cross-lingual transfer accuracy.The paper reports these results as evidence of improvements across model scales and NLU settings.
  • 5 CONCLUSIONS: The authors establish DeBERTaV3 as a new SOTA family for NLU at Large, Base, Small, and XSmall scales.The conclusion also identifies parameter efficiency as an area for further study.

A.1 DATASET

The appendix describes the benchmarks used to evaluate natural language understanding, covering question answering, acceptability, sentiment, similarity, paraphrase detection, inference, commonsense reasoning, named entities, and multilingual inference.

  • GLUE: GLUE aggregates NLU tasks spanning question answering, linguistic acceptability, sentiment analysis, text similarity, paraphrase detection, and natural language inference.The passage describes GLUE as a collection of nine NLU tasks.
  • Reading Comprehension: RACE is a machine reading comprehension dataset drawn from English examinations for middle-school and high-school students.It is described as large-scale.
  • Reading Comprehension: SQuAD v1.1 and v2.0 are reading-comprehension benchmarks built from approximately 500 Wikipedia articles with crowdsourced questions and answers.SQuAD v2.0 additionally includes unanswerable questions about the same paragraphs.
  • Commonsense Reasoning: SWAG evaluates grounded commonsense inference through 113k multiple-choice questions about grounded situations.The dataset combines natural language inference with physically grounded reasoning.
  • Named Entity Recognition: CoNLL-2003 is an English dataset containing text from varied sources and four types of named entities.
  • Multilingual Inference: XNLI provides development and test sets in 15 languages, plus English training data and machine-translated training data for the other 14 languages.

A.2 PRE-TRAINING DATASET

DeBERTaV3 uses the same broad pre-training data as RoBERTa and DeBERTa, while its multilingual version uses CC100 data aligned with XLM-R. The appendix summarizes these datasets and their comparison across models.

  • Pre-training Dataset: English DeBERTaV3 pre-training uses a combination of Wikipedia, BookCorpus, CCNews, Stories, and OpenWebText.
  • Pre-training Dataset: The multilingual DeBERTaV3 model is trained on 2.5TB of CC100 data, the same dataset used for XLM-R.
  • Dataset Comparison: Table 8 compares the pre-training datasets used by different pre-trained models.

A.3 GENERALITY OF GDES

The paper tests whether GDES generalizes beyond DeBERTaV3 by applying it in an ELECTRA setting and evaluating models on MNLI and SQuAD v2.0.

  • GDES is applied to an ELECTRA-style setup with a 12-layer discriminator and 6-layer generator sharing the same hidden size.The models were pretrained from scratch on Wikipedia and BookCorpus for 125k steps with a batch size of 2k.
  • The resulting pretrained models are evaluated on MNLI and SQuAD v2.0 using the setting reported in Table 2.
  • Table 9 reports fine-tuning results for base ELECTRA models trained with different embedding-sharing methods.

A.4 IMPLEMENTATION DETAILS

The implementation largely follows DeBERTa’s pretraining setup while combining MLM-trained generators with RTD-trained discriminators, with hyperparameters documented for pretraining and downstream fine-tuning.

  • The generator uses MLM with 15% of input tokens randomly replaced by [MASK], while the discriminator uses RTD as in ELECTRA.
  • Section 3 experiments use English Wikipedia and BookCorpus with a batch size of 2k for 125k steps.
  • Section 4 experiments use the data listed in Table 8 with a batch size of 8k for 500k steps.
  • The code is implemented based on DeBERTa and ELECTRA.
  • The paper documents DeBERTaV3 pretraining hyperparameters and downstream-task fine-tuning hyperparameters in Tables 10 and 11.
Loading 2111.09543v4…