PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies

Tianyi Zhang, Tatsunori Hashimoto

arXiv:2104.05694v1cs.CLcs.AIcs.LG

TL;DR

The paper asks why generic, task-agnostic masking transfers to downstream tasks when cloze-based accounts offer only a partial explanation. It connects MLM to statistical-dependency learning and evaluates the recovered structure through unsupervised parsing, reporting a higher UUAS than a classic baseline.

  • Problem

    It remains unclear why task-agnostic masking captures linguistic structure and transfers to downstream tasks, while cloze-like explanations do not account for uniform masking’s success.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    The paper relates MLM objectives to graphical-model structure learning and extracts statistical dependencies, evaluating whether they align with syntactic structure.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    Uniform masking performs as well as a control with cloze-like masks removed, and dependency-based parsing scores 58.74 vs 55.91 UUAS against a classic unsupervised parsing baseline.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results indicate that generic masks are associated with direct token dependencies that show a close correspondence to syntactic structures in unsupervised parsing evaluations.

    Source: §6 · p. 8

    The analysis assumes MLM samples faithfully approximate the data-generating distribution; its theoretical setup models latent variables and sparse dependencies with Gaussian variables.

    Source: §4.1 · p. 4, §4.2 · p. 6

Abstract

from arXiv · show

We study how masking and predicting tokens in an unsupervised fashion can give rise to linguistic structures and downstream performance gains. Recent theories have suggested that pretrained language models acquire useful inductive biases through masks that implicitly act as cloze reductions for downstream tasks. While appealing, we show that the success of the random masking strategy used in practice cannot be explained by such cloze-like masks alone. We construct cloze-like masks using task-specific lexicons for three different classification datasets and show that the majority of pretrained performance gains come from generic masks that are not associated with the lexicon. To explain the empirical success of these generic masks, we demonstrate a correspondence between the Masked Language Model (MLM) objective and existing methods for learning statistical dependencies in graphical models. Using this, we derive a method for extracting these learned statistical dependencies in MLMs and show that these dependencies encode useful inductive biases in the form of syntactic structures. In an unsupervised parsing evaluation, simply forming a minimum spanning tree on the implied statistical dependence structure outperforms a classic method for unsupervised parsing (58.74 vs. 55.91 UUAS).

1 Introduction

The introduction asks why task-agnostic masking transfers to downstream tasks, despite theories that explain MLM through cloze-like supervision. It argues that generic masks account for much of this success and proposes statistical dependencies as an explanation.

1 Introduction

  • Cloze reductions recast NLP tasks as fill-in-the-blank prompts, motivating the hypothesis that cloze-like masks provide implicit task supervision.For sentiment, the example is predicting “like” or “hate” in “I [MASK] this movie.”Source: §1 · p. 1
  • Uniform masking performs as well as a negative control that explicitly removes cloze-like masks, so cloze-like masks alone cannot explain its downstream gains.The introduction also reports that cloze-like masks can be helpful.Source: §1 · pp. 1–2
  • The paper relates generic-mask MLM objectives to graphical-model dependency learning and reports unsupervised parsing results of 58.74 vs 55.91 UUAS.The reported comparison is against an unsupervised parsing baseline.Source: §1 · p. 2

2 Related works

Related work motivates cloze-based explanations for pretraining and connects statistical dependencies to word representations, while distinguishing this paper’s focus on MLM objectives and linguistic structure.

2 Related works

  • Cloze reductions have supported zero- and few-shot prediction, relation extraction, and useful representations through implicit downstream supervision.The cited examples include Radford et al. (2019); Brown et al. (2020), Petroni et al. (2019); Jiang et al. (2020), and Saunshi et al. (2020).Source: §2 · p. 2
  • Prior cloze theories do not explain why uniformly random masks provide strong gains; lexicon-based experiments show generic, non-cloze masks contribute substantially.The paper quantifies this gap using task-specific cloze-like masks.Source: §2 · p. 2
  • Word-embedding work links representations to low-rank PMI factorization, whereas this paper draws an analogous MLM connection to graphical-model learning through conditional mutual information.The paper also reports evidence that MLM-learned conditional independence structure is closely related to syntax.Source: §2 · p. 2
  • Probing studies find linguistic information in pretrained representations, but this paper instead examines how the pretraining objective encourages models to extract such structures.The cited probing works include Peters et al. (2018); Tenney et al. (2019); Liu et al. (2019a); Hewitt and Manning (2019); Wu et al. (2020).Source: §2 · p. 2

3 Motivation

The motivation contrasts task-specific cloze supervision with uniform masking, then uses a controlled SST-2 study to show that small departures from ideal cloze masks can hurt transfer. This motivates analyzing what uniform-mask MLM learns instead.

3.1 Problem Statement

  • MLM predicts a masked token from its surrounding context, with a mask distribution determining which tokens are hidden.The input sequence is represented as tokens, and the masked token is replaced by a special [MASK] token.Source: §3.1 · p. 2
  • Perfect cloze masks make the MLM objective approximate the downstream finetuning objective, while uniform masking is framed as modeling more of the full data distribution.In BERT, each token is masked with a fixed uniform probability; the paper asks how such masking yields useful inductive biases.Source: §3.1 · p. 3
  • The paper analyzes MLM as similar to conditional-mutual-information graphical-model structure learning to explain what it learns beyond cloze supervision.The controlled case study motivates looking for a framework that explains generic masks’ success in practice.Source: §3.1 · p. 3, §3.2 · p. 4

3.2 Case Study for Cloze-like Masking

  • The SST-2 case study appends each true label as positive or negative, then varies how often the appended label versus a random review word is masked.The tested cloze-like mask proportions are 0, 20, 40, 60, 80, and 100 percent.Source: §3.2 · p. 3
  • The controlled study splits SST-2 training data in half for pretraining and finetuning, and compares pretrained small transformers with a non-pretrained classifier.The label is omitted from finetuning data.Source: §3.2 · p. 3
  • In SST-2, CLOZE-80% performs worse than CLOZE-100%, and CLOZE-20% does not outperform NOPRETRAIN by much.CLOZE-p% mixes p% cloze-like masks with generic masks; the study averages development accuracy across ten trials.Source: §3.2 · p. 3

4 Analysis

Section 4 reframes uniform-mask MLMs as learning statistical dependencies through graphical-model structure learning. It develops a conditional-mutual-information account and a procedure for extracting dependencies implied by MLMs.

4 Analysis

  • MLM objectives correspond to graphical-model structure learning, implicitly training models to recover statistical dependencies among observed tokens.The paper presents this as an account of how uniform masking can help despite lacking task-specific supervision.Source: §4 · p. 4

4.1 Intuition and Theoretical Analysis

  • In the Gaussian setup, masked regression predicts each coordinate from the others; shared latent variables represent semantic information, while sparse observed-variable dependencies represent syntactic structure.Masked regression behaves like first recovering the latent variable and then predicting the target coordinate, with approximation error scaling with off-diagonal covariance terms.Source: §4.1 · p. 4
  • Unlike the PCA error bound, the masked-regression error bound does not scale with per-coordinate noise, focusing instead on interactions that may capture task-agnostic structures such as syntax.PCA recovers latent information when the latent covariance dominates the observed covariance under the stated eigenvalue condition.Source: §4.1 · p. 5
  • Conditional mutual information can identify direct token dependencies despite latent variables when the remaining context captures enough latent information.The gap between conditional mutual information with and without latent variables is bounded by H(Z|X\{i,j}).Source: §4.1 · p. 5
  • Low MLM log-loss bounds the gap between true conditional mutual information and a model-based estimator, encouraging the model to match the data's conditional dependencies.The MLM objective corresponds to the relevant KL divergence up to a constant entropy term.Source: §4.1 · p. 5

4.2 Extracting statistical dependencies implied by MLMs

  • The extraction procedure estimates conditional mutual information from MLM log probabilities and uses Gibbs sampling to approximate unavailable conditional samples.It assumes MLM samples faithfully approximate the data distribution; the authors report this approximation suffices for testing whether learned conditional independences capture syntax.Source: §4.2 · p. 6

5 Experiment

The experiments test whether task-specific cloze-like masks explain uniform masking’s downstream gains and whether MLM-learned statistical dependencies reflect syntax. Across classification and parsing evaluations, the findings support generic-mask benefits and a correspondence between conditional mutual information and syntactic structure.

5.1 Uniform vs Cloze-like Masking

  • The masking experiment compares uniform masking with CLOZE and NOCLOZE controls, testing whether uniform-mask gains rely on cloze-like masks.CLOZE uses only cloze-like masks; NOCLOZE excludes them.Source: §5.1 · p. 6
  • Second-stage pretraining continues BERT’s MLM updates; evaluation uses three classification datasets, with task lexicons approximating cloze-like masks.SST-2 and AGNews use 20 fine-tuning examples, while Hyperpartisan uses 515 labeled examples; AGNews cloze-like masks use the top 1,000 logistic-regression features.Source: §5.1 · pp. 6–7
  • UNIFORM performs closer to NOCLOZE than CLOZE, while NOCLOZE still outperforms VANILLA, indicating useful gains from generic masks.CLOZE achieves the best accuracy, showing that cloze-like masks can also help.Source: §5.1 · p. 7

5.2 Analysis: Unsupervised Parsing

  • For parsing, pairwise PMI, conditional PMI, and conditional MI scores generate minimum spanning trees, compared with random, linear-chain, and classic unsupervised parsing baselines.Evaluation uses UUAS on 389 Penn Treebank test sentences of length at most 10.Source: §5.2 · p. 7
  • Conditional MI yields parses stronger than random and better than LINEARCHAIN and the classic method, while PMI is near-random and conditional MI exceeds conditional PMI.The evaluation constructs parses by building minimum spanning trees from pairwise dependence scores.Source: §5.2 · pp. 7–8, Table 1 · p. 8
  • On 5,000 training sentences, conditional MI differs from LINEARCHAIN and diverges from dependency grammar on relations such as number and cc.Some dependency-grammar distinctions reflect human conventions that do not necessarily have consensus.Source: §5.2 · p. 8, Table 2 · p. 8
  • In an example parse, conditional MI captures dobj and conj links but misses cc, illustrating differences between statistical dependence and annotated relations.The analysis notes that the resulting errors need not be failures to estimate dependence; dependence can be defined differently.Source: §5.2 · p. 8

6 Discussion and Conclusion

The discussion argues that generic, task-agnostic masks capture direct statistical dependencies that correspond closely to syntactic structures. The findings suggest that learned statistical dependencies can provide implicit supervision for syntax learning.

6 Discussion and Conclusion

  • A substantial part of MLM pretraining gains cannot be attributed to task-specific cloze-like masks; generic masks encourage models to capture direct statistical dependencies.The paper reports that these dependencies closely correspond to syntactic structures in unsupervised parsing evaluations.Source: §6 · p. 8
  • The findings challenge the view that statistical and syntactic dependencies are fundamentally different: better statistical-dependence measures can serve as implicit supervision for syntax.The authors identify emergent syntax learning and improved dependency-learning mask design as directions for future work.Source: §6 · p. 8

A Experimental Details

The appendix gives implementation details for the paper’s experiments, including model training, evaluation, and parsing procedures.

A Experimental Details

  • Section 3.2 uses two-layer transformers with hidden and intermediate sizes of 64, trains for 10 epochs, and applies early stopping based on validation accuracy.Adam uses learning rates of 1e-3 for pretraining and 1e-4 for fine-tuning.Source: A Experimental Details · p. 11
  • For Section 5.1, second-stage pretraining runs for 10 epochs; SST-2 and AGNews results average 20 trials, and Hyperpartisan results average 50.Hyperpartisan evaluation pools its development and test sets because the test set is small and variation is larger.Source: A Experimental Details · p. 11
  • For Section 5.2, conditional MI and conditional PMI use 2,000 Gibbs-sampling steps, and word-piece dependencies are aggregated by their maximum value.The K&M model trains on WSJ sentences of length at most 10; DMV uses annotated POS tags.Source: A Experimental Details · p. 11

B.1 Additional Results in Section 5.2

Additional parsing experiments test conditional MI on longer Penn Treebank sentences and report relation disagreements with LINEARCHAIN.

B.1 Additional Results in Section 5.2

  • On a 200-sentence subsample averaging 24.1 words per sentence, conditional MI parses retain higher quality than the other baselines.The experiment tests sentences longer than 10 words and reports UUAS in Table 4.Source: §B.1 · p. 11, Table 4 · p. 12
  • Table 5 presents a comprehensive list of all relations on which Conditional MI disagrees with LINEARCHINA under a log odds ratio test with p = 0.05.Source: §B.1 · p. 11, Table 5 · p. 12

C Proofs

The proof section restates Proposition 2, which assumes λk > λXX,k+1 and uses V for the first k eigenvectors of Cov(X).

Proof

The proofs bound the gap between conditional mutual information with and without latent variables and analyze masked-regression coefficients using covariance perturbation and regression comparisons.

  • The Proposition 2 proof applies Davis-Kahan to relate subspace-projection error to matrix perturbation, with λk > λXX,k+1 as its condition.It takes Σ = AΣZZA⊤ + ΣXX and Σ′ = AΣZZA⊤; the eigengap of Σ′ is λk, and Weyl’s inequality bounds the kth-eigenvalue perturbation by λXX,k+1.Source: Proof · p. 13
  • For masked regression, the second coefficient term matches β2SLS, so the proof focuses on bounding the first-term error, which scales with off-diagonal terms.Source: Proof · p. 14
  • The gap between conditional mutual information with and without latent variables is bounded by H(Z|X\{i,j}).Source: Proof · p. 15
  • The Proposition 3 proof expands conditional mutual information using the chain rule and bounds terms using KL-divergence convexity and Jensen’s inequality.Source: Proof · p. 15
Loading 2104.05694v1…