Source-linked AI summary
BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model
Alex Wang, Kyunghyun Cho
TL;DR
The paper addresses how BERT’s bidirectional masked language-modeling objective can support sequence modeling and generation. It formulates BERT as a Markov random field language model with pseudo log-likelihood training, derives Gibbs sampling, and finds diverse, fairly fluent generations that are slightly lower-quality than GPT’s.
Problem
BERT predicts masked words from both left and right context, leaving sequence probability evaluation and sentence sampling non-obvious.
Method
The paper formulates BERT as a Markov random field language model trained with pseudo log-likelihood and uses Gibbs sampling for generation.
Results
BERT’s Gibbs-sampling algorithm produces diverse and fairly fluent generations, but compared to GPT, they are more diverse and of slightly worse quality.
Takeaways & Limitations
The formulation enables principled generation from BERT without additional parameters or training.
Takeaways & Limitations
The sampling methods motivate future work because current approaches require running the model over the entire sequence each iteration and handling variable-length sequences more robustly.
Abstract
from arXiv · showhide
We show that BERT (Devlin et al., 2018) is a Markov random field language model. This formulation gives way to a natural procedure to sample sentences from BERT. We generate from BERT and find that it can produce high-quality, fluent generations. Compared to the generations of a traditional left-to-right language model, BERT generates sentences that are more diverse but of slightly worse quality.
1 Introduction
BERT’s bidirectional masked objective makes sequence probabilities and sentence sampling non-obvious. The paper addresses this by formulating BERT as a Markov random field language model trained with pseudo log-likelihood.
- BERT achieves state-of-the-art results across natural language understanding tasks, including constituency parsing and machine translation.
- Masked language modeling predicts each word from both left and right context, unlike traditional next-word language modeling.
- The bidirectional objective leaves unclear how to evaluate full sequence probabilities or sample sentences from BERT.
- The paper identifies BERT as a Markov random field language model trained with pseudo log-likelihood, yielding a Gibbs-sampling procedure.
2 BERT as a Markov Random Field
BERT can be formulated as a Markov random field language model whose token conditionals are tractable despite an intractable exact joint normalization. It uses pseudo log-likelihood learning to train these conditionals, with stochastic approximation reducing computation.
- MRF formulation: BERT represents sequences as fully connected graphs in which each token variable depends on every other token.
- MRF formulation: The MRF assigns a full-graph potential that decomposes into T log-potential terms, defining a probability over complete sequences.
- MRF formulation: Exact maximum likelihood is intractable because the joint distribution’s normalization constant is impractical to compute exactly.
- Conditional distribution: Given all other tokens, each token’s conditional probability is computed with a softmax over the M vocabulary items.
- Training: BERT uses pseudo log-likelihood learning, maximizing each token’s predictability from the remaining tokens rather than the sequence’s joint probability.
- Computational considerations: Computing pseudo log-likelihood remains expensive because each sequence requires |X| forward passes when fθ is computationally heavy.
- Training: Stochastic pseudo log-likelihood learning masks one token at a time, while original BERT masks multiple tokens and predicts them from observed and masked context.
3 Using BERT as an MRF-LM
BERT can be treated as a Markov random field language model that assigns distributions to fixed-length sentences, enabling sentence scoring and sampling. Gibbs sampling provides the natural non-sequential procedure, while sequential alternatives are also explored.
- Using BERT as an MRF-LM: BERT’s MRF-LM framing defines a distribution over sentences of a given length for scoring or generation.This extends BERT beyond parameter initialization for finetuning.
- Using BERT as an MRF-LM: Gibbs sampling repeatedly masks a randomly selected position, samples its token conditionally, and updates the sequence.The initial state is typically an all-mask sequence, though random words or retrieved sentences are also possible.
- Using BERT as an MRF-LM: Multiple Gibbs chains or different initializations are advised because local MCMC proposals can become stuck in a mode.Thinning is also recommended when repeating the procedure many times.
- Using BERT as an MRF-LM: Sequential sampling masks and generates one position per timestep, then substitutes each sampled token into the sequence.After T timesteps, every position has been sampled; an alternative all-position strategy often became trapped in non-fluent sentences.
4 Experiments
The experiments evaluate BERT generations for quality and diversity against GPT and dataset samples. They use non-sequential top-k sampling and assess similarity, fluency, repetition, and n-gram uniqueness.
- Experiments: Quality is measured with BLEU against WikiText-103 and Toronto Book Corpus references and perplexity from a pretrained Gated Convolutional Language Model.Perplexity serves as a rough proxy for fluency.
- Experiments: Diversity is assessed with self-BLEU and the percentage of unique n-grams relative to the data distribution and generation corpus.Higher self-BLEU indicates lower sample diversity, while unique n-gram rates capture novelty.
- Experiments: BERT uses non-sequential sampling from the top k = 100 most frequent words because this produced the most coherent generations.Sequential-sampler generations are reported separately in the appendix.
- Experiments: The study samples 1000 uncased sequences of length 40 from BERT and GPT, alongside dataset samples as a trivial baseline.GPT has approximately the same parameter count as BERT base and generates left to right.
5 Results
BERT generations are more diverse than GPT generations but somewhat lower quality, while human fluency scores for BERT and GPT are broadly similar.
- BERT generations are of worse quality but more diverse than GPT generations.
- BERT corpus-BLEU with the datasets is low, particularly with WT103.
- BERT has lower n-gram overlap with TBC and WikiText-103, roughly matching the overlap between samples from those datasets.GPT shows higher overlap, including more overlap than the datasets have with themselves, suggesting reliance on generic sentences.
- Human fluency means were 2.37 for BERT large, 2.65 for BERT base, and 2.80 for GPT on a four-point Likert scale.Each model contributed 100 samples, and each sentence received ratings from three annotators.
- BERT base and GPT had similar unimodal fluency distributions, while BERT large had a bimodal distribution.BERT base had slightly more non-fluent samples than GPT.
6 Conclusion
The paper presents BERT as a Markov random field language model and uses Gibbs sampling to generate diverse, fairly fluent sentences without additional training.
- BERT can be formulated as a Markov random field language model.
- Gibbs sampling provides a practical generation algorithm that requires no additional parameters or training.
- The experiments produce diverse and fairly fluent generations from BERT.
- Future work should explore samplers that avoid running the model over the entire sequence at each iteration and better handle variable-length sequences.
A Other Sampling Strategies
The paper compares alternative sampling strategies and reports that generating all positions at every step often becomes trapped in non-fluent sentences.
- The authors explored left-to-right sampling and generating all positions at each time step.
- Generating all positions from an all-mask sequence was intended to reduce computation but tended to get stuck in unrecoverable non-fluent sentences.
- Table 4 presents sample generations from the left-to-right strategy using BERT base.