PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders

Fangyu Liu, Ivan Vulić, Anna Korhonen, Nigel Collier

arXiv:2104.08027v2cs.CLcs.AIcs.LG

TL;DR

Off-the-shelf MLMs are not effective universal lexical or sentence encoders without task-specific fine-tuning, often using annotated data. Mirror-BERT turns them into such encoders through self-supervised duplication and augmentation. It reports large gains over off-the-shelf MLMs across languages and domains, with conversion taking under a minute.

  • Problem

    Off-the-shelf MLMs do not produce high-quality lexical or sentence embeddings without task-specific fine-tuning, and can underperform static or specialized encoders.

    Source: §1 · p. 1

  • Method

    Mirror-BERT self-supervises MLM fine-tuning by duplicating and slightly augmenting input text or its representations, without external supervision.

    Source: §1 · p. 2

  • Results

    Mirror-BERT greatly outperforms off-the-shelf MLMs on similarity tasks across languages and domains, and converts MLMs within a minute on two NVIDIA RTX 2080Ti GPUs.

    Source: §1 · p. 2, §4.4 · p. 6

  • Takeaways & Limitations

    The findings support rewiring MLMs into universal encoders without external supervision, with input- and feature-space augmentation providing a synergistic effect.

    Source: §1 · p. 2

    In social media biomedical entity linking, PubMedBERT+Mirror does not match knowledge-guided SapBERT, and harder synonymy relations require external knowledge.

    Source: §4.1 · p. 5

Abstract

from arXiv · show

Pretrained Masked Language Models (MLMs) have revolutionised NLP in recent years. However, previous work has indicated that off-the-shelf MLMs are not effective as universal lexical or sentence encoders without further task-specific fine-tuning on NLI, sentence similarity, or paraphrasing tasks using annotated task data. In this work, we demonstrate that it is possible to turn MLMs into effective universal lexical and sentence encoders even without any additional data and without any supervision. We propose an extremely simple, fast and effective contrastive learning technique, termed Mirror-BERT, which converts MLMs (e.g., BERT and RoBERTa) into such encoders in 20-30 seconds without any additional external knowledge. Mirror-BERT relies on fully identical or slightly modified string pairs as positive (i.e., synonymous) fine-tuning examples, and aims to maximise their similarity during identity fine-tuning. We report huge gains over off-the-shelf MLMs with Mirror-BERT in both lexical-level and sentence-level tasks, across different domains and different languages. Notably, in the standard sentence semantic similarity (STS) tasks, our self-supervised Mirror-BERT model even matches the performance of the task-tuned Sentence-BERT models from prior work. Finally, we delve deeper into the inner workings of MLMs, and suggest some evidence on why this simple approach can yield effective universal lexical and sentence encoders.

1 Introduction

Off-the-shelf masked language models (MLMs) are not reliable universal lexical or sentence encoders without task-specific fine-tuning. Mirror-BERT offers a fast, self-supervised transformation that improves similarity performance across languages and domains.

1 Introduction

  • Off-the-shelf MLMs do not produce high-quality lexical or sentence embeddings without task-specific fine-tuning and can underperform static or specialized encoders.Source: §1 · p. 1
  • Recent universal-encoder approaches use labelled NLI or sentence-similarity data, or knowledge bases, and therefore require external human-labelled resources.Source: §1 · pp. 1–2
  • Mirror-BERT transforms pretrained MLMs into universal lexical and sentence encoders by duplicating and slightly augmenting text, without task-labelled external data.The transformation takes less than one minute on two 2080Ti GPUs.Source: §1 · p. 2
  • Mirror-BERT greatly outperforms off-the-shelf MLMs on similarity tasks across languages and domains, and input- and feature-space augmentation have a synergistic effect.Source: §1 · p. 2

2 Mirror-BERT: Methodology

Mirror-BERT creates positive pairs from duplicated text, augments them in the input or feature space, and uses contrastive learning to shape the encoder’s representations.

2 Mirror-BERT: Methodology

  • The methodology creates positive pairs, augments them in the input or feature space, then applies contrastive learning to attract positives and repel negatives.Source: §2 · p. 2

2.1 Training Data through Self-Duplication

  • Mirror-BERT constructs positive training examples without external data by assigning each input string its own label and duplicating it.The duplicated strings form positive pairs whose representation distance is minimized.Source: §2.1 · p. 2

2.2 Data Augmentation

  • Input-space augmentation masks spans, while feature-space dropout creates different representations of identical strings and can also augment word-level inputs.The authors also evaluate combining span masking with dropout.Source: §2.2 · pp. 2–3

2.3 Contrastive Learning

  • The InfoNCE objective fine-tunes the encoder to bring each self-duplicated positive pair together and push other strings in the batch apart.The negatives for an input are the other strings in its current batch.Source: §2.3 · p. 3

3 Experimental Setup

The setup evaluates lexical, sentence-level, and cross-lingual representations across similarity, biomedical entity-linking, entailment, and bilingual lexicon tasks, using sampled data for Mirror-BERT training.

Lexical evaluation

  • Word similarity and biomedical entity linking cover general and biomedical lexical tasks, with Spearman’s ρ and accuracy @1/@5 as their respective measures.The biomedical evaluation includes NCBI-disease, BC5CDR-disease, BC5CDR-chemical, AskAPatient, and COMETA.Source: §3 · pp. 3–4

Sentence-level evaluation

  • Sentence evaluation uses English and multilingual STS datasets, plus QNLI, assessed as an unsupervised task with development-set AUC from cosine similarity.QNLI uses binary labels on approximately 5.4k development pairs.Source: §3 · p. 4

Cross-lingual evaluation

  • Cross-lingual evaluation covers word similarity and bilingual lexicon induction, with BLI measured by accuracy @1 using CSLS retrieval.The BLI setup uses standard mapping-based training and test sets.Source: §3 · p. 4

Training resources

  • Training samples comprise 10k words per language, 10k UMLS names for biomedical models, and 10k sentences from task-related sources for sentence-level models.STS sentence samples are unlabeled; Spanish, Arabic, and Russian samples come from Wiki-Matrix.Source: §3 · p. 4

Training setup

  • Models use AdamW at 2e-5 with batch size 200 (400 after duplication); lexical models train for two epochs and sentence models for one.The setup also specifies [CLS] representations, dropout 0.1, task-specific temperatures, and random span masking for sentence and biomedical phrase models.Source: §3 · p. 4

4 Results and Discussion

Mirror-BERT improves lexical and sentence encoders across languages and tasks, while analyses show that its performance depends on the base model, data, and augmentation choices. The authors also report that it can be run quickly and may reorganize existing MLM knowledge rather than add new knowledge.

4.1 Lexical-Level Tasks

  • Mirror-BERT substantially improves word-similarity and biomedical entity-linking performance; on Multi-SimLex it beats fastText in 5 of 8 languages.Gains are more pronounced with stronger base MLMs, while PubMedBERT remains stronger than BERT on biomedical datasets.Source: §4.1 · pp. 4–5

4.2 Sentence-Level Tasks

  • On English STS, RoBERTa+Mirror matched SBERT using 10k unlabeled sentences; on QNLI, RoBERTa+Mirror scored .709 versus SBERT’s .706.Mirror-BERT also produced large gains on Spanish, Arabic, and Russian STS.Source: §4.2 · p. 6

4.3 Cross-Lingual Tasks

  • Mirror-BERT produced large gains across cross-lingual word-similarity and bilingual lexicon-induction tasks, though gains were usually smaller for language pairs involving Hebrew.The authors suggest this may reflect mBERT’s weaker semantic knowledge for Hebrew.Source: §4.3 · p. 6

4.4 Further Discussion and Analyses

  • With 10k positive pairs, conversion takes under a minute on two RTX 2080Ti GPUs; performance peaks around 10k–20k examples in the analyzed tasks.Those analyses covered BERT on Multi-SimLex, PubMedBERT on COMETA, and RoBERTa on STS12.Source: §4.4 · p. 6
  • Random span masking and dropout together perform best, while controlled dropout underperforms standard dropout and even no dropout.This comparison suggests the gains stem from feature augmentation rather than regularization alone; alternative masking methods tested were typically similar or worse.Source: §4.4 · pp. 6–7
  • Mirror-BERT reduces overall cosine similarities, separates positive from negative sentence pairs, and increases embedding-space isotropy.Without augmentation, isotropy increased only for word-level representations, leaving the roles of positive and negative gradients unresolved.Source: §4.4 · p. 8
  • Even random strings improved Multi-SimLex English ρ from 0.267 to 0.481, although actual English text performed better.The authors interpret this as evidence that Mirror-BERT rewires knowledge already present in MLMs rather than acquiring new knowledge from the tuning data.Source: §4.4 · p. 8
  • Frequent in-domain examples tended to perform better, and using STS-trained BERT on QNLI lowered AUC from .674 to .665.The authors report similar STS performance when tuning with STS, Wikipedia, or NLI text, though optimal hyperparameters differed.Source: §4.4 · p. 9

5 Related Work

The related work traces self-supervised representation learning from corpus co-occurrence methods to contrastive learning on pretrained language models. Mirror-BERT is distinguished as a lightweight post-hoc approach that also evaluates lexical, biomedical, and cross-lingual settings.

5 Related Work

  • DeCLUTR treats span pairs from the same document as positive examples in contrastive sentence-embedding training.Earlier representation models commonly exploited word, phrase, and sentence co-occurrence statistics.Source: §5 · p. 9
  • Wu et al. (2020) use input augmentation as an auxiliary loss during full MLM pretraining, whereas Mirror-BERT transforms an existing pretrained MLM post hoc.The paper characterizes its approach as a lightweight and fast self-supervised transformation.Source: §5 · p. 9
  • Carlsson et al. (2021) use two models for separate text views, while Mirror-BERT uses dropout and random span masking within one model.The supplied passage describes Mirror-BERT's single-model view generation.Source: §5 · p. 9
  • SimCSE (Gao et al., 2021) shares Mirror-BERT's contrastive loss and dropout augmentation, but is limited to general-domain English sentence representations.The paper also contrasts its broader evaluation with concurrent approaches, covering lexical, biomedical, and cross-lingual tasks.Source: §5 · p. 9

6 Conclusion

The conclusion presents Mirror-BERT as a fast, self-supervised transformation of pretrained MLMs into lexical and sentence encoders, with strong performance across tasks, languages, and domains.

6 Conclusion

  • Mirror-BERT transforms pretrained MLMs into lexical and sentence encoders within a minute, without external supervision, and performs strongly on semantic similarity and biomedical entity linking.The reported gains over base MLMs span different languages and scripts and diverse domains.Source: §6 · p. 9
  • The paper analyzes causes behind Mirror-BERT's efficacy.The conclusion identifies this analysis as part of the work's contribution.Source: §6 · p. 9

A Language Codes

  • Table 11 lists the language abbreviations used in the paper.Source: Table 11

B Additional Training Details

The appendix records training and representation choices, reports run-to-run stability, and identifies which training-set changes may increase fluctuations.

B Additional Training Details

  • The word-level experiments use the 100k most frequent English words in Wikipedia, while the appendix also points to a list of those words.The cited word list is specified as the most frequent 100k English words in Wikipedia.Source: B Additional Training Details · p. 15
  • For Mirror-BERT, [CLS] generally outperforms mean-pooling, except that sentence-level BERT uses mean-pooling; other Mirror-BERT models use [CLS].Representations after the pooling layer did not improve results.Source: B Additional Training Details · p. 15
  • Sentence-level STS standard deviation is below 0.002, while word-level standard deviation is generally below 0.005 across runs.The appendix reports averages over three runs where applicable and describes fine-tuning as generally stable.Source: B Additional Training Details · p. 15
  • Tables 12 and 13 report average STS ρ across dropout-rate and random-span-masking-rate settings, respectively.The captions identify default values with asterisks.Source: Table 12 · p. 15, Table 13 · p. 15
  • Changing the training corpus between runs might lead to larger fluctuations because the sampled training sets were fixed across experiments.Source: B Additional Training Details · p. 15

C Details of Mirror-BERT Trained on Random Strings

The appendix reports that Mirror-BERT trained on random strings can outperform MLM baselines, while comparing its performance with standard word-list training.

C Details of Mirror-BERT Trained on Random Strings

D Dropout and Random Span Masking Rates

Dropout and random span-masking sensitivity differ between BERT and RoBERTa. The reported MVN trends align with isotropy trends under Mirror tuning.

D Dropout and Random Span Masking Rates

E Mean-Vector l2-Norm (MVN)

  • Lower MVN indicates a more mean-centered embedding, and mean-centering is described as increasing spatial isotropy by shifting the space’s center toward the data points.Source: E Mean-Vector l2-Norm (MVN) · p. 16
  • MVN trends align with IS: Mirror tuning makes the space more isotropic and shifts its center closer to the data points’ center.Table 14 reports MVN and IS for word-, phrase-, and sentence-level models; lower MVN and higher IS indicate more isotropic spaces.Source: E Mean-Vector l2-Norm (MVN) · p. 16, Table 14 · p. 16

F Evaluation Dataset Details

The appendix documents evaluation data, pretrained encoders, full results tables, model size, and hyperparameter selection, including limits on per-setting optimality.

F Evaluation Dataset Details

  • Dataset download links are provided in the code repository; the Russian STS dataset is from deepPavlovEval and QQP is downloaded from Kaggle.Source: F Evaluation Dataset Details · p. 16

G Pretrained Encoders

  • For each language, monolingual MLMs were selected by popularity based on download counts, and BASE models were used instead of LARGE for computational tractability.Source: G Pretrained Encoders · p. 16

H Full Tables

  • Full tables cover multilingual and cross-lingual word similarity, Spanish, Arabic, and Russian STS, BLI, ablations, and MVN and IS; MLM features use mean-pooling or [CLS].Source: H Full Tables · p. 16

I Number of Model Parameters

J Hyperparameter Optimisation

K Software and Hardware Dependencies

The appendix specifies the software stack, hardware references, and hyperparameter search grid used for the experiments.

K Software and Hardware Dependencies

Related papers

Loading 2104.08027v2…