BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model

• BERT achieves state-of-the-art results across natural language understanding tasks, including constituency parsing and machine translation.Source: §1 · p. 1
• Masked language modeling predicts each word from both left and right context, unlike traditional next-word language modeling.Source: §1 · p. 1
• The bidirectional objective leaves unclear how to evaluate full sequence probabilities or sample sentences from BERT.Source: §1 · p. 1
• The paper identifies BERT as a Markov random field language model trained with pseudo log-likelihood, yielding a Gibbs-sampling procedure.Source: §1 · p. 1

MRF formulation

• BERT represents sequences as fully connected graphs in which each token variable depends on every other token.Source: §2 · p. 1
• The MRF assigns a full-graph potential that decomposes into T log-potential terms, defining a probability over complete sequences.Source: §2 · p. 1
• Exact maximum likelihood is intractable because the joint distribution’s normalization constant is impractical to compute exactly.Source: §2 · p. 2

Conditional distribution

• Given all other tokens, each token’s conditional probability is computed with a softmax over the M vocabulary items.Source: §2 · pp. 1–2

Training

• BERT uses pseudo log-likelihood learning, maximizing each token’s predictability from the remaining tokens rather than the sequence’s joint probability.Source: §2 · p. 2

Computational considerations

• Computing pseudo log-likelihood remains expensive because each sequence requires |X| forward passes when fθ is computationally heavy.Source: §2 · p. 2

Training

• Stochastic pseudo log-likelihood learning masks one token at a time, while original BERT masks multiple tokens and predicts them from observed and masked context.Source: §2 · p. 2

Using BERT as an MRF-LM

• BERT’s MRF-LM framing defines a distribution over sentences of a given length for scoring or generation.This extends BERT beyond parameter initialization for finetuning.Source: §3 · p. 2
• Gibbs sampling repeatedly masks a randomly selected position, samples its token conditionally, and updates the sequence.The initial state is typically an all-mask sequence, though random words or retrieved sentences are also possible.Source: §3 · p. 3
• Multiple Gibbs chains or different initializations are advised because local MCMC proposals can become stuck in a mode.Thinning is also recommended when repeating the procedure many times.Source: §3 · p. 3
• Sequential sampling masks and generates one position per timestep, then substitutes each sampled token into the sequence.After T timesteps, every position has been sampled; an alternative all-position strategy often became trapped in non-fluent sentences.Source: §3 · p. 3

Experiments

• Quality is measured with BLEU against WikiText-103 and Toronto Book Corpus references and perplexity from a pretrained Gated Convolutional Language Model.Perplexity serves as a rough proxy for fluency.Source: §4.1 · p. 4
• Diversity is assessed with self-BLEU and the percentage of unique n-grams relative to the data distribution and generation corpus.Higher self-BLEU indicates lower sample diversity, while unique n-gram rates capture novelty.Source: §4.1 · p. 4, Table 2 · p. 4
• BERT uses non-sequential sampling from the top k = 100 most frequent words because this produced the most coherent generations.Sequential-sampler generations are reported separately in the appendix.Source: §4.1 · p. 4
• The study samples 1000 uncased sequences of length 40 from BERT and GPT, alongside dataset samples as a trivial baseline.GPT has approximately the same parameter count as BERT base and generates left to right.Source: §4.1 · p. 4

• BERT generations are of worse quality but more diverse than GPT generations.Source: §5 · pp. 4–5
• BERT corpus-BLEU with the datasets is low, particularly with WT103.Source: §5 · p. 5
• BERT has lower n-gram overlap with TBC and WikiText-103, roughly matching the overlap between samples from those datasets.GPT shows higher overlap, including more overlap than the datasets have with themselves, suggesting reliance on generic sentences.Source: §5 · p. 5
• Human fluency means were 2.37 for BERT large, 2.65 for BERT base, and 2.80 for GPT on a four-point Likert scale.Each model contributed 100 samples, and each sentence received ratings from three annotators.Source: §5 · p. 5
• BERT base and GPT had similar unimodal fluency distributions, while BERT large had a bimodal distribution.BERT base had slightly more non-fluent samples than GPT.Source: §5 · p. 5

• BERT can be formulated as a Markov random field language model.Source: §6 · p. 5
• Gibbs sampling provides a practical generation algorithm that requires no additional parameters or training.Source: §6 · p. 5
• The experiments produce diverse and fairly fluent generations from BERT.Source: §6 · p. 5
• Future work should explore samplers that avoid running the model over the entire sequence at each iteration and better handle variable-length sequences.Source: §6 · p. 5

• The authors explored left-to-right sampling and generating all positions at each time step.Source: A Other Sampling Strategies · p. 6
• Generating all positions from an all-mask sequence was intended to reduce computation but tended to get stuck in unrecoverable non-fluent sentences.Source: A Other Sampling Strategies · p. 6
• Table 4 presents sample generations from the left-to-right strategy using BERT base.Source: Table 4 · p. 7