Source-linked AI summary
Enabling Language Models to Fill in the Blanks
Chris Donahue, Mina Lee, Percy Liang
TL;DR
Text infilling predicts missing spans using both preceding and subsequent context, but existing approaches are limited by directionality, fixed span lengths, or specialized architectures. The paper introduces infilling by language modeling, showing that off-the-shelf and pretrained language models can infill effectively, with ILM-generated sentences hardest for humans to recognize as machine-generated.
Problem
Existing language models struggle with general text infilling because unidirectional models lack subsequent context, bidirectional models typically require fixed-length spans, and some variable-length methods use specialized architectures.
Method
Infilling by language modeling trains or fine-tunes language models on concatenations of artificially masked text and the text that was masked.
Results
ILM enables off-the-shelf language models to infill effectively, achieves the highest human-evaluation score, and benefits from large-scale pretrained initialization.
Takeaways & Limitations
The framework may support writing assistance and remain useful in limited-data settings without requiring a specialized architecture.
Takeaways & Limitations
The formulation may redundantly require the model to predict unmasked words and does not guarantee exact reproduction of the unmasked text.
Abstract
from arXiv · showhide
We present a simple approach for text infilling, the task of predicting missing spans of text at any position in a document. While infilling could enable rich functionality especially for writing assistance tools, more attention has been devoted to language modeling---a special case of infilling where text is predicted at the end of a document. In this paper, we aim to extend the capabilities of language models (LMs) to the more general task of infilling. To this end, we train (or fine-tune) off-the-shelf LMs on sequences containing the concatenation of artificially-masked text and the text which was masked. We show that this approach, which we call infilling by language modeling, can enable LMs to infill entire sentences effectively on three different domains: short stories, scientific abstracts, and lyrics. Furthermore, we show that humans have difficulty identifying sentences infilled by our approach as machine-generated in the domain of short stories.
1 Introduction
Text infilling predicts missing spans using both preceding and subsequent context, extending ordinary left-to-right language modeling. ILM adapts existing language models through a simple masked-text training formulation and infills effectively across several domains.
- Text infilling predicts missing spans consistent with both preceding and subsequent text, unlike language modeling, which predicts only after preceding text.
- Ordinary language models generate coherent text efficiently but cannot effectively infill because they use context in only one direction.
- Bidirectional systems can use both sides of a blank, but fixed-length spans or specialized architectures limit their flexibility.
- ILM concatenates artificially masked text with the masked text and trains existing language-model architectures using standard language-model procedures.
- Experiments show that ILM enables off-the-shelf language models to infill effectively, with improvements from large-scale pretraining, across stories, scientific abstracts, and lyrics.
2 Problem Statement
The paper formalizes infilling as completing incomplete token sequences containing one or more blanks. Because multiple completions may be reasonable, the goal is to learn a conditional distribution over completed texts.
- Infilling maps incomplete text ˜x containing one or more missing spans to completed text x.
- An infilling strategy must determine both how many tokens and which tokens to generate for every blank.
- Because a single incomplete text may admit many reasonable completions, the framework learns a distribution p(x | ˜x).
3 Infilling by Language Modeling
ILM reformulates infilling so a unidirectional language model predicts only the missing spans, then deterministically inserts them into the incomplete text. Masked training examples preserve bidirectional context while limiting sequence-length overhead.
- 3.1 Formulation: ILM predicts only missing spans y rather than reproducing unmasked text, then replaces each blank deterministically with the predicted spans.
- 3.1 Formulation: Multiple variable-length spans are represented as one concatenated target separated by special [answer] tokens, yielding p(y | ˜x).
- 3.2 Training Examples: Training masks random spans in complete text, concatenates the masked input, [sep], and the removed spans, and trains an LM on the resulting sequence.
- 3.2 Training Examples: The framework trains or fine-tunes GPT-2 with standard language-model methodology, while allowing other language models in principle.
- 3.3 Advantages: For k missing spans, the training sequence adds only 2k+1 tokens relative to the original, avoiding the effective doubling caused by directly predicting complete text.
- 3.3 Advantages: Because all incomplete text precedes the target, ILM uses context on both sides of blanks while retaining left-to-right language-model decoding.
4 Experimental Setup
The experiments compare ILM with three language-model baselines across stories, scientific abstracts, and lyrics using a common GPT-2 architecture. Models support multiple masking granularities, while evaluation primarily focuses on sentence infilling and pretrained initialization.
- 4.1 Datasets: Experiments evaluate off-the-shelf LM architectures on short stories, scientific abstracts, and song lyrics.
- 4.1 Mask Function: The masking function trains models to infill words, n-grams, sentences, paragraphs, and documents using distinct granularity tokens.
- 4.1 Mask Function: Tokens are masked at about 15% marginal probability by randomly masking subtrees in a granularity hierarchy.
- 4.1 Mask Function: The study primarily evaluates sentence infilling, while results for other granularities are reported in the appendix and exposed in the web demo.
- 4.2 Task and Model Configurations: All strategies use the same GPT-2 small architecture and compare ILM with past-context, future-context, and all-context baselines.
- 4.2 Task and Model Configurations: Pretrained initialization outperforms training from scratch for every infilling strategy, including ILM.
- 4.3 Evaluation: Table 1 reports sentence-infilling test perplexity and relative example length, with lower values preferred for both columns.
5 Quantitative Evaluation
The quantitative evaluation tests sentence infilling with perplexity across datasets and finds that ILM uses bidirectional context effectively while preserving language-modeling performance.
- Perplexity is computed only over the original masked sentence, because overlap metrics such as BLEU are unsuitable for realistic infills.
- Across all datasets, ILM outperforms models using only past or future context, showing that unidirectional models can exploit bidirectional context.
- ILM achieves similar performance to LM-All while using shorter sequences and therefore much less memory.LM-All concatenates the original and masked text, producing longer training sequences.
- ILM-trained models perform similarly on language modeling compared with models trained only on language modeling.This supports retaining the original language-modeling functionality while adding infilling capability.
6 Human Evaluation
The human evaluation asks annotators to identify machine-generated sentences in short stories. ILM receives the highest naturalness score, making its infills hardest to recognize as machine-generated.
- The evaluation replaces one of five human-written story sentences with a model output and asks annotators to identify the generated sentence.The study collected 100 responses for each model.
- The score measures the percentage of examples where annotators failed to identify the machine-generated sentence, so higher values indicate more human-like outputs.
- ILM achieves the highest score among the four strategies, indicating that humans have the most difficulty recognizing its infilled sentences as fake.The score is the percentage of stories in which annotators failed to identify the machine-generated sentence.
- The maximum score is effectively 80%, because a perfect model would make annotators choose randomly among five sentences.
- In the qualitative example, BERT and SA produce off-topic sentences, LM ignores future context, while ILM and Human account for both preceding and subsequent context.
7 Related Work
Prior infilling methods often constrain span length, require iterative reprocessing, or target narrower settings. ILM instead supports variable-length infilling through a language-modeling framework.
- Many existing systems cannot automatically determine span length and therefore infill only fixed-length spans.
- Some methods fill multiple variable-length sequences but iteratively update and reprocess the masked context for each blank.
- ILM appends infilled text to the context without reprocessing the entire input for every blank.
- AI21’s language model fills the middle of a paragraph from its first and last sentences, whereas ILM generalizes this capability.
- Unlike story-generation systems conditioned on titles, entities, premises, or storylines, this work predicts from surrounding text alone.
8 Conclusion
The paper presents a simple language-model-based strategy for text infilling. It reports effective sentence infilling, human-indistinguishable outputs in short stories, and potential relevance to writing assistance.
- The proposed strategy leverages language models to perform text infilling.
- ILM can infill sentences that humans have difficulty recognizing as machine-generated.
- The framework remains effective when initialized from large-scale pre-trained language models, which may help in limited-data settings.
- Future work will incorporate these capabilities into co-creation systems that assist humans with writing.
A Datasets
The experiments use three distinct text domains and a hierarchical masking framework that supports infilling at multiple granularities. The implementation trains existing language models with standard language-modeling procedures on artificially masked examples.
- Datasets: The datasets comprise 100K short-story examples, 200K scientific abstracts, and 2M song-lyrics examples.STORIES contains 5M words, ABSTRACTS 30M words, and LYRICS 60M words.
- Datasets: STORIES uses ROCStories, ABSTRACTS uses computer-science arXiv papers, and LYRICS uses song lyrics from lyrics.com.Each ROCStories story contains a title and five sentences.
- Datasets: The experiments span multiple datasets and include metadata paragraphs that support conditional generation and implicit summarization.Examples include infilling a title given a story or a story given a title.
- Masking: The mask function selects spans across words, n-grams, sentences, paragraphs, and entire documents, allowing users to specify infilling granularity.This provides coarse control over output length and reduces the risk of generating more text than intended.
- Masking: Masking is applied through a document tree with 3% subtree-selection probability, while selected words are masked individually half the time or as random n-grams otherwise.N-grams range from 1 to min(8, # words left in the sentence) words.
- Training and extensibility: The framework adds granularity-specific special tokens and supports user-defined mask functions for different infilling use cases.Training uses standard language-modeling procedures, with early stopping based on validation perplexity and a batch size of 24.
- Training and extensibility: Training maximizes likelihood over the concatenated masked input, separator, and target, including auxiliary supervision on the fully specified masked input.The authors report that this additional supervision improved validation perplexity for the target spans.
D Evaluation on language modeling and infilling other granularities
Evaluation compares ILM with forward, reverse, and all-context language-modeling baselines across language modeling and multiple infilling granularities. ILM generally improves over one-sided baselines, matches LM-All in several settings, and uses less memory.
- Evaluation setup: The evaluation studies GPT-2 from scratch and from a pre-trained checkpoint across language modeling, mixed granularities, and individual granularities.The individual-granularity evaluations cover paragraphs, sentences, n-grams, and words across three datasets.
- Language modeling: Document infilling is equivalent to language modeling because the input is always [blank document].The document-masking setup reserves 3% of examples for this language-modeling case.
- Language modeling: ILM achieves performance similar to LM on document infilling despite seeing far fewer language-modeling examples.This indicates that ILM can retain standard language-modeling ability while supporting infilling.
- Mixture of granularities: Across mixed granularities, ILM outperforms LM and LM-Rev and is similar to LM-All despite using much less memory.The evaluation masks multiple variable-length spans and computes perplexity on the masked spans.
- Individual granularities: Across individual granularities, ILM outperforms LM and LM-Rev and either outperforms or is comparable with LM-All while using less memory.Perplexity is computed on identical masked-span token sets for all models.
E Details on human evaluation
Human evaluation tests sentence infilling in short stories by asking annotators to identify machine-generated completions among outputs from eight models. The procedure includes controls, a human reference, and qualitative and quantitative reporting.
- Evaluation design: The study masks one sentence in each of 100 test stories, producing 500 masked-story contexts for model evaluation.Each model is tasked with infilling the masked sentence.
- Evaluation design: Eight models are compared, including BERT, LM, ILM, scratch-trained variants, a spam-control model, and the original human sentence.The human reference provides a sanity check for the evaluation.
- Annotation procedure: Each annotator sees eight stories and identifies the one machine-generated sentence, with five responses removed after incorrect control-model annotations.The remaining quantitative and qualitative results are reported in Table 9 and Figure 5.
- Reported outputs: Table 9 reports the human-evaluation results, while Figure 5 provides examples of sentence-level infills from different models.Figure 4 illustrates the task and instruction shown to Mechanical Turk annotators.