PaperlayerView exact revision on arXiv

Source-linked AI summary

Blank Language Models

Tianxiao Shen, Victor Quach, Regina Barzilay, Tommi Jaakkola

arXiv:2002.03079v2cs.CLcs.LG

TL;DR

Text infilling must generate missing fragments at controlled locations and variable lengths while respecting surrounding context. BLM dynamically creates and fills blanks in partial or blank-only text. Across infilling, restoration, and style transfer it shows superior performance, achieving 92.5% accuracy on Amazon sentiment transfer.

  • Problem

    Text infilling requires generating missing fragments at specified locations with variable lengths while respecting preceding and following context.

    Source: §1 · p. 1

  • Method

    BLM iteratively selects a blank, inserts a word, and optionally creates adjacent blanks, stopping when no blanks remain.

    Source: §1 · p. 1, §3 · p. 2

  • Results

    Across text infilling, ancient text restoration, and style transfer, BLM shows superior performance; it achieves 92.5% accuracy on Amazon sentiment transfer.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    The demonstrated applications include text infilling, ancient text restoration, and style transfer, supporting BLM as a framework for flexible text generation.

    Source: §5 · p. 9

    The naive lower-bound training procedure has large variance and computes only one action loss per pass, unlike left-to-right models’ n word losses.

    Source: §3 · p. 4

Abstract

from arXiv · show

We propose Blank Language Model (BLM), a model that generates sequences by dynamically creating and filling in blanks. The blanks control which part of the sequence to expand, making BLM ideal for a variety of text editing and rewriting tasks. The model can start from a single blank or partially completed text with blanks at specified locations. It iteratively determines which word to place in a blank and whether to insert new blanks, and stops generating when no blanks are left to fill. BLM can be efficiently trained using a lower bound of the marginal data likelihood. On the task of filling missing text snippets, BLM significantly outperforms all other baselines in terms of both accuracy and fluency. Experiments on style transfer and damaged ancient text restoration demonstrate the potential of this framework for a wide range of applications.

1 Introduction

Text infilling requires controlled completion of variable-length missing spans while respecting surrounding context. BLM dynamically fills blanks and demonstrates effectiveness across text infilling, ancient restoration, and style transfer.

1 Introduction

  • Text infilling must handle any number of missing spans and unknown numbers of missing tokens while controlling location and respecting both surrounding contexts.The introduction identifies text editing, information fusion, and ancient text restoration as tasks requiring generation from partially specified text.Source: §1 · p. 1
  • BLM uses a special blank symbol to control token placement and iteratively fills blanks, optionally adjoining new blanks, until none remain.It can start from a single blank or partial text with specified blanks, and is trained by maximizing a lower bound on likelihood marginalized over trajectories.Source: §1 · p. 1
  • BLM shows superior performance in text infilling, ancient text restoration, and style transfer; it achieves 92.5% accuracy on Amazon sentiment transfer.For restoration of ancient text that lost half its characters, it reduces the error rate by 3.3 points compared to previous methods.Source: §1 · pp. 1–2

2 Related Work

Prior rewriting approaches adapt language models or sequence-generation methods, but often constrain insertion locations, lengths, or generation order. BLM is framed for flexible completion of missing text anywhere in an input.

2 Related Work

  • The Insertion Transformer supports dynamic word insertion but does not let users specify insertion locations or distinguish intact spans from scattered missing parts.The passage says directly forcing it to perform infilling can lead to suboptimal solutions.Source: §2 · p. 2
  • For rewriting, masked language models require insertion lengths and heuristic mask-generation order, while XL-Net does not support unknown-length infilling.The cited methods are Fedus et al. (2018), Wang and Cho (2019), Ghazvininejad et al. (2019), Yang et al. (2019), and Shih et al. (2019).Source: §2 · p. 2
  • Unlike generation-from-scratch methods tailored to a specific order, BLM is designed to complete missing parts located anywhere in input text.The passage states that BLM can dynamically accommodate insertions of various lengths.Source: §2 · p. 2

3 Blank Language Models

BLM generates text by repeatedly selecting a blank, filling it with a word, and optionally creating adjacent blanks. Its training objective sums over generation trajectories, while its architecture factorizes the action into location, word, and blank-creation decisions.

3 Blank Language Models

  • A BLM action selects a blank, inserts a word, and decides whether to create blanks to its left and right, allowing a blank to expand to any number of words.Generation begins with a single blank and ends when no blanks remain.Source: §3 · p. 2
  • The architecture encodes the canvas, chooses among current blank positions, predicts a word, then uses an MLP to decide whether adjacent blanks are created.The blank representation and chosen word embedding are concatenated for the final decision.Source: Figure 3 · p. 3, §3 · p. 3
  • BLM can be viewed as a grammar in which blanks are nonterminals and vocabulary items are terminals, with production probabilities conditioned on the entire canvas.Its production rules replace a blank with a word and optionally preceding or following blanks.Source: §3 · pp. 2–3
  • Because a sentence can arise from multiple trajectories, BLM trains by maximizing a Jensen lower bound on likelihood that encourages equal realization across generation orders.Directly computing the marginal likelihood over n! orders is intractable.Source: §3 · pp. 3–4
  • The efficient training algorithm combines loss calculations for trajectories sharing the same initial steps and computes n/2 action losses per pass in expectation.The naive lower-bound estimate has large variance and computes only one action loss per pass, unlike left-to-right models’ n word losses.Source: §3 · p. 4

4 Experiments

Experiments assess BLM on text infilling, ancient-text restoration, sentiment transfer, and language modeling, measuring both rewriting outcomes and language-model perplexity.

4.1 Text Infilling

  • BLM has the highest BLEU across mask ratios, by 0.7–1.7 over InsT, 2.6–4.1 over MLM (oracle length), and 3.7–9.4 over BERT+LM.The comparison is reported for text-infilling documents; Table 1 covers BLEU and perplexity.Source: Table 1 · p. 5, §4.1 · p. 5

4.2 Ancient Text Restoration

  • L-BLM matches Pythia’s single-slot CER and beats Pythia-Word by 4.4 and 3.3 points at 40% and 50% masking.The authors posit that joint likelihood over all slots explains the advantage over Pythia-Word, which considers one slot at a time.Source: Table 3 · p. 7, §4.2 · p. 7

4.3 Sentiment Transfer

  • BLM infilling raises accuracy over MLM by 38.1% on Yelp and 20.8% on Amazon; classifier-guided inference reaches 92.5% Amazon accuracy with a small BLEU decrease.The inference-time classifier selects the highest-ranked of 10 sampled outputs.Source: §4.3 · pp. 7–8, Table 4 · p. 8

4.4 Language Modeling

  • BLM rivals Insertion Transformer and outperforms LSTM- and TCN-based left-to-right language models; Transformer-XL and adaptive embeddings achieve the top results.Perplexity is estimated by Monte Carlo sampling; the estimate is expected to exceed actual perplexity, and m = 1000 is reported as close to convergence.Source: Table 6 · p. 8, §4.4 · pp. 8–9

5 Conclusion

BLM fills specified blanks with variable-length text consistent with context, using actions that select and expand blanks. It is trained over possible generation trajectories and demonstrated on several text-rewriting tasks.

5 Conclusion

  • BLM fills one or more specified blanks with a variable number of tokens consistent with context.Its actions select a blank and replace it with a word, possibly adjoining new blanks.Source: §5 · p. 9
  • BLM is demonstrated on text infilling, ancient text restoration, and style transfer.The paper also identifies template filling, information fusion, and assisting human writing as potential applications.Source: §5 · p. 9
  • Training optimizes a lower bound on marginal data likelihood summed over generation trajectories, encouraging sentences to be realized in different orders.The model can also be trained to favor specific orders by assigning those trajectories higher weights.Source: §5 · p. 9

A.1 Insertion Transformer

The Insertion Transformer baseline is implemented in the authors’ framework with the BLM Transformer encoder and the baseline’s own prediction and canvas-generation mechanisms.

A.1 Insertion Transformer

  • The implementation uses the same Transformer encoder module as BLM.Source: §A.1 · p. 12
  • Its prediction layers are replaced with the Insertion Transformer mechanism.Source: §A.1 · p. 12
  • The canvas is generated according to the Insertion Transformer training procedure.Source: §A.1 · p. 12

A.2 Masked Language Model

The appendix describes baseline configurations for masked-language-model, BERT-plus-language-model, and sequence-to-sequence comparisons. The masked-model baseline receives target-length mask sequences at test time, while the sequence-to-sequence baselines use blanked training copies.

A.2 Masked Language Model

  • At test time, the masked-language-model baseline receives mask sequences expanded to the target length, equivalent to an oracle supplying infilling length.Among one-shot, left-to-right, and confident-first decoding, confident-first performs best and is reported.Source: §A.2 · p. 12

A.3 BERT+LM

  • The BERT+LM baseline concatenates BERT’s output at each blank position with the left-to-right decoder’s input word embedding.Source: §A.3 · p. 12

A.4 Seq2seq-full and Seq2seq-fill

  • Both sequence-to-sequence baselines use Fairseq’s transformer iwslt de en architecture and are trained with k blanked copies per sentence; k = 10 gives the best reported performance.Source: §A.4 · p. 12

B Monte-Carlo Estimate of Perplexity

The appendix estimates sentence likelihood using randomly sampled generation orders and uses that estimate to derive perplexity. The likelihood estimate is unbiased, while the expected estimated perplexity is at least the actual perplexity and tightens as sample count grows.

B Monte-Carlo Estimate of Perplexity

C Generation Trajectory

Figure 7 presents examples of BLM generation trajectories on the Yelp review dataset.

C Generation Trajectory

  • Figure 7 shows example BLM generation trajectories on Yelp reviews.Source: Figure 7 · p. 13

Related papers

Loading 2002.03079v2…