ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators

1 INTRODUCTION

• MLM selects a small subset of tokens, typically 15%, for prediction, so the network learns from only that subset per example and incurs substantial compute cost.Source: §1 · p. 1
• Replaced token detection corrupts input with plausible generator-sampled replacements and trains a discriminator to classify every token as original or replaced.Unlike MLM, the task learns from all input tokens rather than only the masked subset.Source: §1 · p. 1
• At equal model size, data, and compute, ELECTRA substantially outperforms MLM methods; ELECTRA-Large matches RoBERTa and XLNet using 1/4 of their training compute.It also outperforms ALBERT on GLUE and sets a new state-of-the-art for SQuAD 2.0.Source: Figure 1 · p. 2, §1 · p. 2
• Although the discriminator resembles a GAN discriminator, the generator is trained with maximum likelihood rather than adversarially.The paper notes that adversarial training is difficult for text because sampling prevents backpropagation.Source: §1 · p. 2
• Ablations show that learning from all input positions makes ELECTRA train faster than BERT, while fully trained ELECTRA achieves higher downstream accuracy.Source: §1 · p. 2

2 METHOD

• The generator is usually a small masked language model trained jointly with the discriminator, though it can be any model that produces a token distribution.Source: Figure 2 · p. 3
• The discriminator learns to distinguish original tokens from generator replacements in a corrupted input, while the generator is trained with masked language modeling.The generator predicts masked-out tokens; the discriminator evaluates tokens in the resulting corrupted example.Source: §2 · p. 3
• The input is corrupted by replacing selected masked positions with samples from the generator rather than leaving [MASK] tokens in place.Typically, k = ⌈0.15n⌉ tokens are masked out.Source: §2 · p. 3
• A generator-sampled token that matches the original is labeled “real,” and this formulation moderately improved downstream results.The generator is trained with maximum likelihood rather than adversarially because sampling prevents back-propagation through the generator.Source: §2 · p. 3
• After pre-training, the generator is discarded and the discriminator is fine-tuned on downstream tasks.Source: §2 · p. 4

3.1 EXPERIMENTAL SETUP

• Experiments use GLUE and SQuAD, mostly with 3.3 billion Wikipedia and BooksCorpus tokens, and use 33 billion tokens for ELECTRA-Large; fine-tuned scores are medians of 10 runs.The evaluation is on English data; the Large-model comparison is not entirely direct because its pre-training data differs from RoBERTa’s.Source: §3.1 · p. 4, §3.4 · p. 7

3.2 MODEL EXTENSIONS

• 1/4–1/2-size generators worked best; tied embeddings raised GLUE from 83.6 without tying to 84.3 with token tying and 84.4 with all-weight tying, while joint training beat two-stage and adversarial training.Tying all weights requires generator and discriminator to be the same size; two-stage training improved after switching to the discriminator objective but did not beat joint training.Source: §3.2 · pp. 4–6, Figure 3 · p. 5

3.3 SMALL MODELS

• 5 GLUE points higher than comparable BERT-Small, ELECTRA-Small also outscored the much larger GPT; base-sized ELECTRA outperformed BERT-Large’s 84.0 GLUE score.ELECTRA-Small used the same training FLOPs as BERT-Small; ELECTRA’s small-model gains held when trained on a single GPU.Source: Table 1 · p. 6, §3.3 · p. 6

3.4 LARGE MODELS

• With less than 1/4 of RoBERTa and XLNet’s pre-training compute, ELECTRA-400K performs comparably on GLUE; on SQuAD, it beats RoBERTa-100k and matches RoBERTa-500K using less than 1/4 the compute.ELECTRA-1.75M outscores RoBERTa and XLNet on most GLUE tasks while still requiring less pre-training compute.Source: Table 2 · p. 7, §3.4 · p. 7, Table 4 · p. 8

3.5 EFFICIENCY ANALYSIS

• ELECTRA 15% performs much worse than ELECTRA, Replace MLM slightly outperforms BERT, and All-Tokens MLM closes most of the gap; most of ELECTRA’s improvement is attributed to learning from all tokens.The authors attribute a smaller part of the improvement to alleviating the pre-train/fine-tune mismatch from [MASK] tokens.Source: §3.5 · pp. 8–9, Table 5 · p. 9
• Even when fully trained, small ELECTRA models reach higher downstream accuracy than BERT, and ELECTRA’s gains grow as model size decreases.The authors speculate that ELECTRA may be more parameter-efficient, while noting that further analysis is needed to explain this.Source: Figure 4 · p. 9, §3.5 · p. 9

Self-Supervised Pre-training for NLP

• BERT pre-trains a Transformer with masked language modeling, while XLNet predicts tokens autoregressively in a random order but generates only 15% of input tokens.The authors note that XLNet may avoid BERT’s [MASK] discrepancy, though its two attention streams during pre-training and one during fine-tuning leave this unclear.Source: §4 · p. 9

Generative Adversarial Networks

• ELECTRA resembles GAN-discriminator approaches but does not use adversarial learning; text GAN methods still lag maximum-likelihood training.The passage also relates ELECTRA’s generator to MaskGAN, which fills tokens deleted from the input.Source: §4 · p. 10

Contrastive Learning

• ELECTRA is related to Noise-Contrastive Estimation, which also trains a binary classifier to distinguish real from fake data points.Source: §4 · p. 10
• ELECTRA can be viewed as a scaled-up CBOW with negative sampling, but uses a Transformer and a learned generator instead of a bag-of-vectors encoder and unigram proposal.Source: §4 · p. 10

• Replaced token detection trains an encoder to distinguish input tokens from high-quality negative samples produced by a small generator network.Source: §5 · p. 10
• Compared with masked language modeling, the objective is more compute-efficient and yields better downstream-task performance.Source: §5 · p. 10
• The authors hope the approach will make pre-trained text encoders more accessible and encourage reporting compute usage and parameter counts alongside evaluation metrics.Source: §5 · p. 10

• Both ELECTRA and BERT baselines mostly use BERT hyperparameters, with dynamic token masking and no next-sentence-prediction objective.The stated rationale for omitting next-sentence prediction is that recent work suggested it does not improve scores.Source: A PRE-TRAINING DETAILS · p. 13
• ELECTRA-Large uses a 25% mask rate rather than 15% because the generator achieved high accuracy at 15%, leaving very few replaced tokens.Source: A PRE-TRAINING DETAILS · p. 13
• For Base and Small models, the authors searched learning rates from [1e-4, 2e-4, 3e-4, 5e-4] and selected λ from [1, 10, 20, 50, 100] in early experiments; otherwise, they did no tuning beyond Section 3.2.Source: A PRE-TRAINING DETAILS · p. 13

• The authors searched fine-tuning epochs for Large models and learning rates and layer-wise decay for Base models, while keeping settings shared across tasks except for epochs.They expect results might improve slightly with the task-specific hyperparameter search used in prior GLUE work.Source: B FINE-TUNING DETAILS · p. 13
• The authors omit WNLI dev results because the standard fine-tuning-as-classifier approach is difficult to beat even with a majority classifier.Source: B FINE-TUNING DETAILS · p. 13
• For GLUE test results, RTE and STS use intermediate training from an MNLI-fine-tuned checkpoint, while WNLI scores candidate antecedents with the discriminator and each task ensembles 10 of 30 seeds.For RTE, intermediate training is combined with a 2e-5 learning rate.Source: B FINE-TUNING DETAILS · pp. 13–14
• Leaderboard tricks improve scores but make scientific comparisons harder because they require extra implementation and compute and differ across papers.ELECTRA-1.75M is also reported using only dev-set model selection, the setting BERT used for its results.Source: B FINE-TUNING DETAILS · pp. 14–15
• For the SQuAD 2.0 test submission, the authors fine-tuned 20 models from one pre-trained checkpoint and submitted the model with the best dev score.Source: B FINE-TUNING DETAILS · p. 15

C DETAILS ABOUT GLUE

• GLUE covers eight tasks: acceptability, sentiment, paraphrase, similarity, question answering, and textual entailment or inference.The tasks include CoLA, SST, MRPC, STS, QQP, MNLI, QNLI, and RTE.Source: C DETAILS ABOUT GLUE · p. 15
• Training-set sizes range from 2.5k RTE examples to 393k MNLI examples, with data drawn from sources including books, reviews, news, and Wikipedia.Other listed datasets range from 3.7k MRPC examples to 364k QQP examples.Source: C DETAILS ABOUT GLUE · p. 15
• GLUE task metrics vary: STS uses Spearman correlation, CoLA uses Matthews correlation, and the other tasks use accuracy.This appendix supplies descriptions and dataset details for the benchmark tasks.Source: C DETAILS ABOUT GLUE · p. 15

D FURTHER RESULTS ON GLUE

• ELECTRA-Base++ and ELECTRA-Small++ use XLNet data and 4e6 training steps; Small++ also uses sequence length 512, and both ++ models score higher.ELECTRA-Small++ otherwise uses the Table 6 hyperparameters.Source: D FURTHER RESULTS ON GLUE · p. 15
• ELECTRA appears particularly effective on CoLA, where identifying fake tokens may resemble distinguishing linguistically acceptable sentences from ungrammatical ones.The proposed explanation is tentative in the passage.Source: D FURTHER RESULTS ON GLUE · p. 15
• ELECTRA-Base outperforms BERT-Large and ELECTRA-Small outperforms GPT in GLUE average score; Table 8 reports single-task fine-tuning results without ensembling or task-specific tricks.The ++ variants perform even better, but the supplied text gives no specific scores.Source: D FURTHER RESULTS ON GLUE · p. 15, Table 8 · p. 16

E COUNTING FLOPS

• Compute is measured in hardware-agnostic FLOPs, counting mathematical operations rather than machine instructions and matrix multiplication as 2*m*n FLOPs.The estimates assume equal forward- and backward-pass FLOPs and dense embedding lookups; the text notes these assumptions are imperfect.Source: E COUNTING FLOPS · p. 16

F ADVERSARIAL TRAINING

• Because discrete sampling prevents backpropagation through the discriminator, generator training uses policy-gradient reinforcement learning.The appendix identifies this adversarial setup as Adversarial Contrastive Estimation rather than Generative Adversarial Training.Source: F ADVERSARIAL TRAINING · pp. 16–17
• The generator is trained to maximize discriminator loss, while non-replaced-token scores are excluded because the assumption makes them independent of the generator.The generator takes a single action representing simultaneous token generation.Source: F ADVERSARIAL TRAINING · pp. 16–17
• The generator predicts all tokens simultaneously, and the simplifying assumption makes each discriminator prediction depend only on the token and non-replaced context.This removes dependence on other generated tokens and lets the generator objective be decomposed over individual generated tokens.Source: F ADVERSARIAL TRAINING · pp. 16–17
• The adversarially trained generator reaches 58% masked-language-modeling accuracy versus 65% for the same-sized maximum-likelihood generator, and does not improve downstream performance.The adversarial generator has hidden size 256.Source: F ADVERSARIAL TRAINING · p. 17

G EVALUATING ELECTRA AS A MASKED LANGUAGE MODEL

• For a fixed generator, the appendix derives the discriminator's optimal form and proposes using it to select ELECTRA's masked-token prediction.The prediction is selected by maximizing a function of discriminator and generator probabilities over the vocabulary.Source: G EVALUATING ELECTRA AS A MASKED… · pp. 17–18
• BERT achieves 77.9% accuracy versus ELECTRA's 75.5% on masked-language modeling, although the assumed optimal discriminator may impair ELECTRA's evaluation accuracy.The comparison uses Wikipedia+BooksCorpus; vocabulary selection is restricted to the generator's top 100 predictions for practicality.Source: G EVALUATING ELECTRA AS A MASKED… · p. 18
• The proposed evaluation selects among the generator's top 100 predictions rather than the whole vocabulary because full-vocabulary selection is very expensive.The passage frames BERT's advantage as unsurprising given its generation-focused training and ELECTRA's discriminative objective.Source: G EVALUATING ELECTRA AS A MASKED… · p. 18

H NEGATIVE RESULTS

• Strategically masking tokens produced fairly minor speedups over regular BERT.Strategies included masking rarer tokens more often or predicting which tokens BERT would struggle to reconstruct.Source: H NEGATIVE RESULTS · p. 18
• Raising the generator softmax temperature or preventing it from sampling the correct token did not improve results.These changes were explored after ELECTRA appeared to benefit from a weaker generator up to a point.Source: H NEGATIVE RESULTS · p. 18
• The sentence-level contrastive objective slightly decreased downstream task scores.The experiment kept 20% of input sentences unchanged and added a head to predict whether the entire input was corrupted.Source: H NEGATIVE RESULTS · p. 18