PaperlayerView exact revision on arXiv

Source-linked AI summary

COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining

Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, Xia Song

arXiv:2102.08473v2cs.CLcs.LG

TL;DR

COCO-LM addresses diminishing pretraining signals and problematic Transformer representation geometry by correcting and contrasting auxiliary-model-corrupted sequences. Its corrective language modeling and sequence contrastive learning objectives improve GLUE and SQuAD effectiveness while reducing pretraining compute, including matching ELECTRA’s MNLI accuracy with 50% of its GPU hours.

  • Problem

    Self-supervised pretraining faces increasingly uninformative signals, limited language-modeling capability in binary classification, and anisotropic sequence representations.

  • Method

    COCO-LM uses an auxiliary model to corrupt text, then applies Corrective Language Modeling for token detection and correction and Sequence Contrastive Learning for alignment and uniformity.

  • Results

    COCO-LM outperforms recent state-of-the-art models on GLUE and SQuAD while matching ELECTRA’s MNLI accuracy with 50% of its pretraining GPU hours.

  • Takeaways & Limitations

    COCO-LM improves accuracy and uses pretraining computing resources and network parameters more efficiently with standard data and Transformer architectures.

  • Takeaways & Limitations

    ELECTRA-style binary classification lacks language-modeling capability needed for some tasks, including prompt-based learning.

Abstract

from arXiv · show

We present a self-supervised learning framework, COCO-LM, that pretrains Language Models by COrrecting and COntrasting corrupted text sequences. Following ELECTRA-style pretraining, COCO-LM employs an auxiliary language model to corrupt text sequences, upon which it constructs two new tasks for pretraining the main model. The first token-level task, Corrective Language Modeling, is to detect and correct tokens replaced by the auxiliary model, in order to better capture token-level semantics. The second sequence-level task, Sequence Contrastive Learning, is to align text sequences originated from the same source input while ensuring uniformity in the representation space. Experiments on GLUE and SQuAD demonstrate that COCO-LM not only outperforms recent state-of-the-art pretrained models in accuracy, but also improves pretraining efficiency. It achieves the MNLI accuracy of ELECTRA with 50% of its pretraining GPU hours. With the same pretraining steps of standard base/large-sized models, COCO-LM outperforms the previous best models by 1+ GLUE average points.

1 Introduction

COCO-LM addresses inefficiency and irregular sequence representations in self-supervised pretraining by correcting corrupted tokens and contrasting related sequences. It improves benchmark effectiveness and pretraining efficiency over recent approaches.

  • Motivation: Standard token-level pretraining becomes less informative over time, making further effectiveness gains increasingly compute- and parameter-intensive.The passage describes this scaling pattern as unsustainable.
  • Motivation: Pretrained sequence representations can be anisotropic, with irregular geometry that complicates sequence-level applications.Such representations may require dedicated fine-tuning approaches.
  • Approach: Binary replaced-token classification improves efficiency but can hinder language-modeling applications and distort representations by assigning actual tokens the same label.These limitations motivate COCO-LM’s corrective and contrastive objectives.
  • Approach: COCO-LM corrupts input texts with an auxiliary model and trains the main Transformer with Corrective Language Modeling and Sequence Contrastive Learning.CLM detects and corrects replaced tokens, while SCL aligns sequences from the same source and enforces representation uniformity.
  • Results: 50% of ELECTRA’s pretraining GPU hours suffices for COCO-LM to match ELECTRA’s MNLI accuracy.With the same pretraining steps, COCO-LM also exceeds previous best models by 1+ GLUE average points.

2 Related Work

Prior work developed token-level and sequence-level pretraining objectives, with MLM remaining broadly effective. ELECTRA introduced efficient replaced-token detection, while later work explored contrastive learning and related alternatives.

  • Token-level pretraining: Autoregressive, masked, permutation, and pseudo-masked language modeling pretrain Transformers by recovering tokens from contextual input.These approaches differ in how context and prediction order are constructed.
  • Sequence-level pretraining: Sequence-level objectives commonly classify concatenated sentence pairs based on relationships such as adjacency, ordering, or prior-sentence context.Examples include next sentence prediction, sentence ordering, and previous sentence prediction.
  • Comparative evidence: MLM remains among the most effective general pretraining tasks, whereas other objectives show more task-specific benefits.Reported specialized benefits include entity-related masking for knowledge-intensive tasks and sequence objectives for long-form modeling.
  • Corruption-based methods: ELECTRA uses an auxiliary MLM generator to replace tokens and trains a main discriminator to detect them, while ELECTRIC contrasts original tokens with cloze-model negatives.These methods motivate corruption-based and contrastive pretraining designs.
  • Contrastive learning: Contrastive learning has shown strong results in visual representation learning and has been increasingly applied to language during fine-tuning.Applications include sentence representation, dense retrieval, and GLUE fine-tuning.

3 Method

COCO-LM extends ELECTRA-style pretraining with corrective language modeling and sequence contrastive learning to preserve language-modeling benefits and improve representation geometry.

  • Challenges: ELECTRA-style classification can omit language-modeling capability, limit word-level semantic capture, and produce squeezed representations with poor alignment and uniformity.RoBERTa random pairs cluster near 0.8, while ELECTRA random and similar pairs cluster around 0.9 cosine similarity.
  • Framework: COCO-LM uses an MLM-pretrained auxiliary Transformer to generate corrupted sequences for the main Transformer’s pretraining tasks.The auxiliary model is discarded after pretraining, while the main model is used downstream.
  • Corrective Language Modeling: CLM trains the main Transformer to recover original tokens from corrupted sequences using a language-modeling probability rather than only a binary classification score.Its copy mechanism identifies tokens that can be copied directly, while other tokens are corrected from the vocabulary.
  • Corrective Language Modeling: CLM combines all-token binary classification with language modeling on masked positions, preserving ELECTRA’s efficiency and language-modeling capability.The binary classification task helps the model learn the harder language-modeling task through a stop-gradient copy mechanism.
  • Sequence Contrastive Learning: SCL contrasts corrupted sequences with randomly cropped contiguous spans from the same original input while treating other batch sequences as negatives.The cropped span retains 90% of the original sequence length, preserving its major meaning for positive pairing.
  • Sequence Contrastive Learning: SCL encourages representations robust to token-level replacements and sequence-level cropping by aligning corrupted and cropped views of the original information.The objective encourages reasoning from partially altered sequences to recover original information.

4 Experimental Setup

COCO-LM is evaluated under base, base++, and large++ pretraining settings, with specified corpora, model sizes, baselines, and GLUE/SQuAD fine-tuning protocols.

  • Pretraining Settings: Three settings—base, base++, and large++—define the experimental scale, data, training duration, and Transformer configuration.Base follows BERTBase; base++ and large++ use 160 GB of text and 4 billion samples.
  • Pretraining Settings: Base pretraining uses Wikipedia and BookCorpus, 16 GB of text, 256 million samples, 512-token sequences, and a 32,768-token uncased BPE vocabulary.
  • Pretraining Settings: Base++ and large++ use OpenWebText, CC-News, and STORIES, totaling 160 GB, with 4 billion samples and a 64,000-token cased BPE vocabulary.Large++ uses the BERTLarge configuration, while base++ uses the base-size model.
  • Model Architecture: The base models use 12-layer, 768-hidden-size Transformers, whereas large++ uses 24 layers and 1,024 hidden units; auxiliary networks are shallower.Auxiliary networks have 4 layers for base/base++ and 6 layers for large++; dropout is disabled during XMLM generation.
  • Downstream Tasks: Evaluation covers GLUE and SQuAD 2.0, using standard fine-tuning hyperparameter search and median results from five random seeds.
  • Baselines: RoBERTa and ELECTRA are compared as baselines, including models pretrained and fine-tuned under exactly the same settings as COCO-LM.Other baseline numbers come from previously reported results unless marked “(Ours)”.

5 Evaluation Results

COCO-LM outperforms prior pretraining models on GLUE and SQuAD while improving pretraining efficiency, with complementary gains from CLM and SCL. Analyses attribute these gains to better corruption handling, more regularized representations, and effective design choices.

  • Overall Results: COCO-LM outperforms recent state-of-the-art pretraining models on GLUE average and SQuAD across all three evaluation settings.It improves the state-of-the-art GLUE score by about one point under each setting.
  • Efficiency: COCO-LM reaches RoBERTa’s and ELECTRA’s MNLI accuracy with around 60% and 50% of their pretraining GPU hours, respectively.With the same GPU hours, it exceeds both baselines by 1+ MNLI points.
  • Ablation Studies: Combining CLM and SCL provides better overall effectiveness than either task alone, with CLM stronger on MNLI-mm and SCL stronger on RTE and MRPC.Both tasks provide additional improvements over the RTD-only backbone on MNLI and GLUE average.
  • Analyses of Contrastive Learning with SCL: Cropping the original sequence improves SCL training, whereas no cropping is less informative and retaining only 70% can alter semantics excessively.A simple alteration empirically works best for forming positive SCL pairs.
  • Analyses of Contrastive Learning with SCL: SCL makes similar sequence representations more aligned and random sequence representations more uniformly distributed, without task-specific data or supervised labels.Many similar pairs approach cosine similarity 1, while random pairs center around 0.
  • Analyses of Contrastive Learning with SCL: With 1% MNLI labels, SCL improves MNLI-m/mm accuracy by 0.8/0.5 over pretraining without SCL, with larger gains when fewer labels are available.The experiments use standard fine-tuning with fractions of the MNLI training set.
  • Analyses of Language Modeling with CLM: CLM addresses the difficulty of modeling corrupted sequences by combining correction with copying, while All-Token MLM creates an unbalanced task dominated by original tokens.Only 7−8% of tokens are replaced, requiring detection and recovery of the original token.
  • Analyses of Language Modeling with CLM: COCO-LM outperforms RoBERTa in prompt-based MNLI fine-tuning despite seeing no [MASK] tokens during pretraining.ELECTRA and COCO-LM variants without CLM are not applicable because their main Transformers lack learned language-modeling capability for generating prompt label words.

6 Conclusions and Future Work

COCO-LM improves GLUE and SQuAD accuracy while using pretraining resources and parameters more efficiently. Its limitations include simple contrastive-pair construction and an auxiliary model that is not tailored to the main model.

  • COCO-LM improves accuracy on the GLUE and SQuAD benchmarks while using pretraining computing resources and network parameters more efficiently.
  • The contrastive pairs rely on simple cropping and MLM replacements, motivating exploration of more advanced construction methods.
  • The auxiliary Transformer is trained independently of the main Transformer and does not learn to generate its optimal pretraining signals.

A GLUE Tasks

GLUE comprises nine language-understanding tasks spanning inference, equivalence, sentiment, acceptability, paraphrase, and semantic similarity, with varied datasets and evaluation settings.

  • Inference: MNLI and RTE evaluate textual entailment or contradiction judgments between premise–hypothesis pairs.MNLI contains 393K training examples, while RTE contains 2.5K.
  • Equivalence: QQP and MRPC determine whether question or sentence pairs are semantically equivalent.QQP has 364K training examples, while MRPC has 3.7K.
  • Question answering: QNLI predicts whether a sentence contains the answer to a question sentence.The dataset contains 108K training examples derived from SQuAD.
  • Sentiment: SST-2 classifies sentences as positive or negative according to sentiment annotations from movie reviews.The dataset contains 67K training examples.
  • Acceptability: CoLA evaluates whether sentences are linguistically acceptable.Its 8.5K training examples come from books and journal articles on linguistic theory.
  • Similarity: STS-B predicts sentence-pair semantic similarity on a 1 to 5 scoring scale.The dataset contains 5.8K training examples with human annotations.

B Hyperparameter Settings

The experiments retain mostly default pretraining hyperparameters, specify task-specific corruption and contrastive settings, and use matched evaluation procedures across methods.

  • The auxiliary MLM uses the standard 15% [MASK] ratio, while cropping removes 10% of each original sequence.The crop transformation therefore produces a sub-sequence that is 90% of the original length.
  • The SCL softmax temperature is 1, and COCO-LM pretraining tasks receive equal weights except λcopy = 50.
  • All methods use the same or equivalent hyperparameters for fair comparison, with per-task fine-tuning searches and median results from five seeded runs.

C The Origins of Reported Baseline Scores

Reported baseline results primarily come from the original papers, with specified exceptions drawn from other cited studies and the highest score selected when reports differ.

  • Baseline results come from original papers except for specified BERT, RoBERTa, ELECTRA, and XLNet results taken from cited alternative sources.
  • When multiple papers report different scores for one method, the comparisons use the highest reported score.

D More Implementation Details

COCO-LM’s implementation balances pretraining cost with several settings that improve performance or stability. Fine-tuning procedures specify searched hyperparameter ranges for GLUE and SQuAD.

  • Pretraining and Fine-tuning Costs: A COCO-LM pretraining run takes about 20 hours in base, two-three weeks in base++, and three-four weeks in large++ settings on 256 V100 GPUs.The SCL task adds one forward and backward pass on the cropped sequence, while CLM has cost similar to ELECTRA.
  • MLM Mode for Corrective Language Modeling: Disabling dropout in the auxiliary MLM during replacement-token sampling slightly improves downstream performance by generating more stable challenging tokens.The sampled tokens are then corrected by the main Transformer.
  • Projection Heads: Removing the projection layer from the main model’s CLM head improves training stability.The auxiliary MLM retains the standard BERT/RoBERTa projection layer, but the main CLM model uses only its final linear layer.
  • Masking Special Tokens for Auxiliary Model Training: Following RoBERTa, the auxiliary model masks special tokens as well as real tokens, slightly improving performance on some tasks.This differs from BERT, which masks only real tokens.
  • Fine-tuning: Tables 7 and 8 report the hyperparameter ranges searched for fine-tuning on GLUE and SQuAD.GLUE is divided into small tasks—CoLA, RTE, MRPC, and STS-B—and large tasks—MNLI, QQP, QNLI, and SST-2.

E More Discussions on PLM Research

The discussion emphasizes that PLM research is constrained by prohibitive computation costs and noisy evaluation practices. The paper responds by improving comparison robustness through varied training-time reporting and public release of code and models.

  • Computational Cost: PLM research faces prohibitive computation costs, making thorough studies difficult because new designs require expensive, lengthy pretraining experiments.The paper frames this as a tension between PLMs’ broad downstream influence and their costly development cycles.
  • Evaluation Robustness: GLUE averages can be unreliable because its small tasks exhibit high variance, weakening comparisons between pretrained language models.The paper argues that evaluation pipelines should more systematically assess whether a new PLM is better than previous models.
  • Reproducibility: The paper reports downstream performance across different training times and releases code and models to support reproducible comparisons under limited computing budgets.These measures are intended to improve the robustness of observations and facilitate future PLM research.
Loading 2102.08473v2…