PaperlayerView exact revision on arXiv

Source-linked AI summary

Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little

Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, Douwe Kiela

arXiv:2104.06644v2cs.CLcs.LG

TL;DR

The paper asks whether MLM success depends on classical syntactic abstractions or higher-order distributional information. It tests this by pre-training RoBERTa with disrupted or absent word order and evaluating downstream and probing performance. The results largely support a distributional account, while probe discrepancies motivate more challenging evaluations.

  • Problem

    The paper investigates whether MLM success depends on learning grammatical abstractions or higher-order distributional information.

  • Method

    The authors pre-train RoBERTa models on word-order-permuted data, remove positional embeddings, and compare them with regularly trained models.

  • Results

    Shuffled-order pre-training has surprisingly little effect on downstream performance after fine-tuning, while permuted models can nearly match unpermuted models on some parametric syntactic probes.

  • Takeaways & Limitations

    The findings support interpreting MLM success as largely driven by higher-order distributional statistics rather than discovery of a classical NLP pipeline.

  • Takeaways & Limitations

    The authors leave open whether the phenomenon is more pronounced in English than in other languages and how it extends to other tasks.

Abstract

from arXiv · show

A possible explanation for the impressive performance of masked language model (MLM) pre-training is that such models have learned to represent the syntactic structures prevalent in classical NLP pipelines. In this paper, we propose a different explanation: MLMs succeed on downstream tasks almost entirely due to their ability to model higher-order word co-occurrence statistics. To demonstrate this, we pre-train MLMs on sentences with randomly shuffled word order, and show that these models still achieve high accuracy after fine-tuning on many downstream tasks -- including on tasks specifically designed to be challenging for models that ignore word order. Our models perform surprisingly well according to some parametric syntactic probes, indicating possible deficiencies in how we test representations for syntactic information. Overall, our results show that purely distributional information largely explains the success of pre-training, and underscore the importance of curating challenging evaluation datasets that require deeper linguistic knowledge.

1 Introduction

The paper asks whether MLMs succeed by learning classical syntactic structure or mainly by modeling distributional information. Pre-training with disrupted word order still produces strong downstream and probing performance, challenging the classical-pipeline explanation.

  • Motivation: The authors disentangle syntactic and distributional explanations by removing word-order information during MLM pre-training.They argue that a sophisticated NLP pipeline would presumably depend on syntactic information conveyed by word order.
  • Implications: The findings suggest that MLM success is largely explained by a distributional prior rather than by reproducing the classical NLP pipeline.The paper presents stronger evaluations as necessary for testing syntax, generalization, and compositionality.
  • Method: RoBERTa models are pre-trained on word-order-permuted corpora while preserving varying amounts of distributional information.The study also removes positional embeddings and trains a unigram-distribution baseline.
  • Results: Pre-training on permuted data has surprisingly little effect on downstream performance after fine-tuning on non-shuffled data.The result holds across a broad set of downstream settings, making unnatural-language pre-training closer to standard MLM pre-training than expected.
  • Results: Permuted models perform well on parametric syntactic probes, sometimes nearly matching unpermuted models, despite non-parametric probes detecting worse syntax-dependent performance.This contrast raises questions about whether parametric probes meaningfully distinguish representations trained with corrupted versus normal word order.

2 Related Work

Related work studies word-order sensitivity, randomization controls, synthetic pre-training, and the validity of probing tasks. Prior results motivate testing whether order information must be acquired during pre-training rather than fine-tuning.

  • Sensitivity to word order in NLU: Earlier studies evaluate information order through sentence ordering, readability, and dialogue coherence tasks.This work shifts attention from sentence order to word order in downstream language understanding.
  • Sensitivity to word order in NLU: Several studies report high accuracy on permuted examples across inference, paraphrase, and sentiment tasks.These studies mainly used models exposed to normal word order during pre-training, leaving the source of order information unresolved.
  • Sensitivity to word order in NLU: The paper extends prior work by testing models that lack natural word order during pre-training.This isolates whether downstream tasks require order information to be ingrained in the pre-trained model.
  • Randomization ablations: Random sentence encoders, reordered inputs, and partially randomized transformers have served as surprisingly strong baselines or probes.These findings reinforce the broader use of randomization ablations for assessing what models actually learn.
  • Synthetic pre-training: Synthetic pre-training studies show that structured non-linguistic data can provide useful priors for downstream language or vision tasks.Papadimitriou and Jurafsky report benefits from MIDI music or Java code despite no vocabulary overlap with target languages.
  • On the utility of probing tasks: Debate over probing motivates comparing parametric probes with more targeted non-parametric tests.The paper treats probe design as central to interpreting apparent syntactic knowledge.

3 Approach

The study pre-trains RoBERTa-base models on natural, permuted, resampled, and position-free data, then evaluates them across language-understanding and probing tasks. Its controls vary access to word order while retaining or removing co-occurrence information.

  • Models: The experiments use the RoBERTa-base MLM architecture because of its computational efficiency and downstream task performance.The authors expect similar insights from other MLM variants with comparable characteristics.
  • Models: All experiments use the 16GB BookWiki corpus, with MN denoting the model trained on its original unmodified text.The corpus combines the Toronto Books Corpus and English Wikipedia.
  • Models: Sentence-level permutation randomly reorders every sentence while preserving words from the same sentence and eliminating words remaining in their original positions.The study also constructs variants that preserve selected n-gram information before permutation.
  • Models: The MUG baseline samples unigrams according to corpus frequencies, removing sentence-level co-occurrence information while retaining word or subword information.Named entities are treated as unigrams in constructing this baseline.
  • Models: The MNP baseline removes positional embeddings from RoBERTa, making the model entirely order agnostic during pre-training.MRI is randomly initialized and measures what can be learned from task fine-tuning with only the model’s base inductive bias.
  • Evaluation: Evaluation covers GLUE, PAWS, and parametric and non-parametric probing tasks.PAWS uses high-lexical-overlap paraphrase and non-paraphrase pairs generated through controlled word swapping.

4 Downstream task results

Across GLUE and PAWS, shuffled pre-training often preserves downstream performance, while shuffled fine-tuning exposes task-specific dependence on word order. The results support distributional information as a major explanation of MLM performance, while highlighting positional embeddings and dataset artifacts.

  • 4.1 Word order permuted pre-training: M1 trails MN by 3.3 points on average across accuracy-based tasks, but remains within 5 points on PAWS.M1 is pre-trained on completely shuffled sentences, whereas MN uses natural text.
  • Experimental scope: The experiments are limited to the 16GB BookWiki corpus and 100K updates, so MN is not directly comparable with the publicly released RoBERTa-base model.The restriction mirrors RoBERTa ablations and was imposed for computational reasons.
  • 4.1 Word order permuted pre-training: M1 outperforms MNP by a large margin, indicating that positional embeddings are critical even when pre-training word orders are unnatural.MNP lacks the ability to learn positional embeddings, while M1 uses shuffled sentences.
  • 4.2 Word order permuted fine-tuning: Shuffled fine-tuning preserves above 80% accuracy on MNLI, SST-2, and MRPC, while QQP and QNLI decline only slightly.These results suggest that purely lexical information is useful for several downstream tasks.
  • 4.2 Word order permuted fine-tuning: All datasets besides QQP and QNLI show noticeable drops when shuffled training is evaluated on natural-order text, especially CoLA, RTE, and PAWS.The pattern suggests these tasks use word order and that shuffled models can acquire it during fine-tuning.

5 Probing results

The paper evaluates syntactic representations with parametric and non-parametric probes. Randomized models often retain high parametric-probe accuracy, while non-parametric probes more clearly reveal losses from removing word order.

  • Probing setup: The authors use parametric probes with learnable parameters and non-parametric probes that directly examine language-model predictions.The evaluation includes dependency parsing, SentEval tasks, and grammaticality-based comparisons.
  • Experimental protocol: The experiments report mean test accuracy over three seeds and examine UD EWT and PTB dependency-parsing datasets.UD EWT contains 12,543 training sentences and PTB contains 39,832.
  • Syntactic probing: Dependency-parsing UAS follows the fine-tuning ordering M1≈MUG< M2< M3< M4< MN, with task- and probe-specific performance gaps.The lowest gaps include MN versus M3/M4 on PTB with both MLP and Linear probes.
  • SentEval probes: Parametric SentEval probes show that randomized models can achieve reasonably high accuracy on syntax-sensitive tasks despite lacking natural word order.MN scores higher on some tasks, but it scores lower than randomized models on two of the three syntactic tasks.
  • Non-parametric probing: The authors question whether parametric probes reliably distinguish models trained with corrupted word order from normally trained models.They introduce non-parametric probes because the interpretation of parametric probing remains debated.
  • Non-parametric probing: Non-parametric probes compare model probabilities for grammatically correct versus incorrect focus words in controlled sentence sets.The datasets include agreement attractors, random same-part-of-speech substitutes, and minimal grammaticality pairs; some datasets are rebalanced for singular and plural focus words.

6 Discussion

The discussion argues that high downstream and parametric-probe performance does not establish that MLMs learn a classical syntactic pipeline. It instead raises questions about what current evaluations measure and about the scope of distributional learning.

  • Interpretation: Fine-tuning and parametric-probing results suggest that MLMs need not rely much on word order to achieve high accuracy.This challenges claims that MLMs learn a classical NLP pipeline whose syntactic information depends on word order.
  • Interpretation: Lower non-parametric probing accuracy indicates that randomized models do not accurately reconstruct the original word order.The discussion distinguishes reconstructing likely orders from representing the actual order required by syntax.
  • Interpretation: Even if models infer word order from unordered text, that inference would be based on distributional information and could recover only the most likely order.The authors emphasize that syntax concerns the actual order rather than merely possible or probable orders.
  • Implications: The results suggest revisiting the meaning of linguistic structure and whether most NLP tasks require human-like linguistic abilities.This is presented as one interpretation of the findings rather than as a universal conclusion.
  • Implications: More challenging and comprehensive evaluations may be needed to measure linguistic abilities in NLP models.The authors frame this as a consequence if evaluations are intended to test linguistic abilities rather than distributional performance.
  • Limitations and future work: The study leaves open whether the phenomenon is stronger in English than in other languages and how it extends to tasks such as extractive QA or sequence tagging.The authors identify cross-linguistic and task-expansion questions as future work.

7 Conclusion

The paper argues that MLM success is largely explained by distributional information rather than learning the syntactic and semantic mechanisms of traditional NLP pipelines.

  • The authors investigate whether MLM success depends on discovering syntax and semantics needed for a traditional language-processing pipeline.
  • The paper connects BERT-style MLMs to earlier distributional models by progressively replacing local contexts and dot products with sentence-wide masked contexts and self-attention.
  • BPE reduces the softmax vocabulary, while additional data, compute, and fine-tuning updates complete the practical BERT recipe.
  • The paper provides a sentence-randomization algorithm that tokenizes words, samples non-overlapping n-grams, conjoins them, and permutes the resulting sequence.
  • The study measures how much local n-gram structure survives sentence randomization using BLEU-2, BLEU-3, and BLEU-4 scores on one million sentences.

C Pre-training details

RoBERTa models are pre-trained on BookWiki variants and evaluated on Wiki-103 perplexity, with stronger word-order randomization producing worse language-modeling scores.

  • The experiments pre-train RoBERTa-base models on variants of the 16GB BookWiki corpus using Fairseq and modified training limits.
  • Figures 2 and 3 report Wiki-103 perplexity and GLUE results for the model variants and ablations.
  • Wiki-103 validation and test sets provide a quick, effective, and reproducible basis for comparing the pre-trained models.
  • Perplexity monotonically increases from MN through M4–M1, then MUG, and finally MNP.

D Word-order pre-training ablations

Ablation results indicate that some distributional prior is needed for strong downstream representations, while shuffled pre-training can remain effective despite reduced word-order information.

  • M512 scores worse overall than M1 but remains significantly better than the MNP and MUG baselines on GLUE.
  • Destroying natural word frequencies in MUF produces comparable or slightly better results than the random-corpus MUG baseline.
  • These ablations indicate that merely reproducing unigram frequencies without context is insufficient, whereas at least some distributional prior supports good downstream representations.
  • Using different randomization seeds for each corpus shard produces minimal downstream differences between M1 and M1*, except for CoLA.
  • The relative-difference metric normalizes each model’s performance gap between natural-order pre-training and random performance, approaching zero at the natural-model level.
  • CoLA has the largest relative difference across tasks, while QQP has the lowest.

F Fine-tuning with randomized data

Fine-tuning on natural data allows models pretrained with randomized word order to approach naturally pretrained performance, although word order remains useful for several tasks and affects early learning.

  • Fine-tuning with randomized data: 94.49 to 62.22: PAWS suffers the largest performance drop on shuffled testing, while gaining least from a unigram-randomized prior.This supports the conclusion that PAWS learns much of its required word-order information during task fine-tuning.
  • Syntactic probes: In dependency parsing, randomized pretraining produces a small performance gap because the complex parser learns most task behavior and uses only a 100-dimensional BERT representation.Published results with and without BERT also show a minimal gap.
  • Perplexity analysis: The naturally pretrained model has the lowest perplexity on natural-order sentences, whereas randomized models prefer their corresponding n-gram randomizations.The result indicates that pretraining models retain sensitivity to the word-order distribution used during pretraining.
  • Fine-tuning with randomized data: Natural pretraining consistently yields the lowest MDL, with the largest early advantage on word-order-reliant datasets such as RTE, CoLA, and PAWS.With more fine-tuning, models can relearn correct word order, reducing the practical impact of the pretraining difference.

J At what point do models learn word order during pre-training?

Task performance rises rapidly during early RoBERTa pretraining, while natural word-order information becomes useful only later for some tasks.

  • At what point do models learn word order during pre-training?: Rissanen Data Analysis measures early fine-tuning difficulty through minimum description length, with lower MDL indicating better learning ability.The analysis compares natural and unnatural pretrained representations on downstream tasks.
  • At what point do models learn word order during pre-training?: Performance on all tasks increases rapidly during the first 20-25 pretraining epochs.The comparison uses intermediate natural-order RoBERTa checkpoints against randomized-pretraining models.
  • At what point do models learn word order during pre-training?: For some tasks, word-order information begins helping only after 30-50 pretraining epochs.Thus, the timing of word-order benefits varies across tasks rather than appearing uniformly during pretraining.

K More results from Syntactic Probes

Additional syntactic-probe results show minimal differences between natural and randomized representations on several probing tasks, while probability-based diagnostics expose instability in some evaluations.

  • More results from Syntactic Probes: Minimal Pareto Hypervolume differences appear among natural and randomized models on dependency parsing across both UD EWT and PTB.The metric is computed as area under the complexity-performance curve across hyperparameter runs.
  • More results from Syntactic Probes: The gap narrows further on POS and DAL, with M1 only 3.5 points below MN on average for PTB in both tasks.These tasks are simpler than dependency parsing in the reported probing framework.
  • Non-parametric probes: Raw accuracy can be misleading when all token probabilities are extremely low, because weak models may receive unexpectedly high scores despite low individual probabilities.The authors therefore report probability differences alongside accuracy.
  • Non-parametric probes: Natural-order pretraining produces the largest probability difference between correct and incorrect focus words, while the difference rises marginally as fewer n-grams are perturbed.The probability-difference measure is scaled by 100 in the plotted results.
  • Non-parametric probes: Singular/plural imbalance inflated weak-model scores, so the authors balanced the stimuli by upsampling inflections.The original unbalanced results are documented separately from the corrected evaluation.
Loading 2104.06644v2…