PaperlayerView exact revision on arXiv

Source-linked AI summary

Text Infilling

Wanrong Zhu, Zhiting Hu, Eric Xing

arXiv:1901.00158v2cs.CLcs.AIcs.LGstat.ML

TL;DR

Text infilling fills missing portions of text, but prior work focused on single-word gaps or gaps at the end of a text. This paper studies arbitrary numbers of unknown-length gaps with self-attention and segment-aware positions; self-attention outperforms seq2seq and GAN-based models across the reported settings and datasets.

  • Problem

    Text infilling is useful, but prior work restricted gaps to single words or a single end-of-text continuation rather than arbitrary numbers of unknown-length missing portions.

    Source: §1 · pp. 1–2

  • Method

    The paper studies supervised text infilling with self-attention, segment-aware positions, and generation conditioned on known and already generated tokens.

    Source: §1 · p. 2, §3.1 · p. 2, §3.2 · p. 3

  • Results

    Across varying mask rates and blank counts and on two long-content datasets, self-attention outperforms seq2seq and GAN-based models in the reported evaluations.

    Source: §4.1 · p. 3, §4.2 · p. 4, §5 · p. 4

  • Takeaways & Limitations

    The authors present self-attention as a strong baseline for future text-infilling research.

    Source: §1 · p. 2

    The study assumes supervised template–filled-text pairs, and leveraging Devlin et al. (2018)’s pretrained model is left for future work.

    Source: §3.1 · p. 2, §1 · p. 2

Abstract

from arXiv · show

Recent years have seen remarkable progress of text generation in different contexts, such as the most common setting of generating text from scratch, and the emerging paradigm of retrieval-and-rewriting. Text infilling, which fills missing text portions of a sentence or paragraph, is also of numerous use in real life, yet is under-explored. Previous work has focused on restricted settings by either assuming single word per missing portion or limiting to a single missing portion to the end of the text. This paper studies the general task of text infilling, where the input text can have an arbitrary number of portions to be filled, each of which may require an arbitrary unknown number of tokens. We study various approaches for the task, including a self-attention model with segment-aware position encoding and bidirectional context modeling. We create extensive supervised data by masking out text with varying strategies. Experiments show the self-attention model greatly outperforms others, creating a strong baseline for future research.

1 Introduction

Text infilling is useful in practical settings but had received less study than other text-generation tasks, with previous work restricting the number or placement of missing tokens. This paper studies arbitrary missing spans and evaluates several approaches, finding that self-attention performs well and can serve as a baseline.

1 Introduction

  • Earlier infilling work assumed either a single missing word per portion or a single missing portion at the text’s end, limiting applicability to multiple randomly placed gaps.The paper identifies such cases as common in real scenarios.Source: §1 · p. 1
  • The task allows any number of missing portions, each containing an arbitrary unknown number of tokens, and fills them using global and surrounding context.Examples include restoring damaged documents, writing with templates, and editing text.Source: §1 · p. 1
  • A self-attention model with segment-aware position encoding is proposed to capture global and surrounding context and localize multiple gaps with varying lengths.The encoding combines segment identity with token offset.Source: §1 · p. 2
  • Automatic and human evaluations find that self-attention performs reasonably well and can serve as a strong baseline for future research.Experiments use randomly or schematically masked text with varying numbers of segments and missing ratios.Source: §1 · p. 2
  • The authors leave leveraging the pretrained model from Devlin et al. (2018) for text infilling to future work.The concurrent work uses a similar model and training objective for text representation learning while focusing on text generation.Source: §1 · p. 2

2 Related Work

Related work covers sequence-to-sequence and adversarial approaches to text generation, while distinguishing this paper’s general infilling setting from restricted prior tasks. It also connects the proposed self-attention approach to concurrent work on bidirectional text representations.

2 Related Work

  • The paper studies a simplified GAN-based method related to MaskGAN (Fedus et al., 2018), whose setting assumes one missing token per blank rather than an arbitrary number.Generalizing MaskGAN to this task is left for future work.Source: §2 · p. 2
  • Sequence-to-sequence models with attention (Sutskever et al., 2014; Bahdanau et al., 2014; Luong et al., 2015) are among the popular approaches to text generation.The paper also situates adversarial training among recent efforts.Source: §2 · p. 2
  • The best-performing approach uses self-attention (Vaswani et al., 2017), resembling a Transformer encoder with bidirectional context.Concurrent work by Devlin et al. (2018) learns text representations by reconstructing randomly masked tokens; the paper describes infilling as a generalization of that task.Source: §2 · p. 2

3 Text Infilling

The task fills deleted or redacted portions of a text template to produce coherent text; the number and positions of blanks are known, but their lengths are not. The paper explores sequence-to-sequence, GAN-based, and self-attention solutions, with self-attention generating each blank token by token and using segment-aware positions.

3.1 Problem Definition

  • Given a template with deleted or redacted portions, the task is to fill the blanks to produce complete, semantically coherent, meaningful text.The number and positions of blanks are known, but the number of tokens to generate for each blank is not.Source: §3.1 · p. 2
  • Training is supervised with template and filled-text pairs, while the model must decide how many tokens to generate for each known blank.The token count for each blank is not provided with the input template.Source: §3.1 · p. 2

3.2 Preliminary Solutions

  • The paper explores attentional seq2seq, GAN-based, and self-attention methods as representative solutions.The GAN adds a discriminator over seq2seq to encourage global coherence.Source: §3.2 · pp. 2–3
  • Self-attention can model both left- and right-side context, unlike sequential left-to-right attention.The authors propose this as a way to encode global semantics effectively.Source: §3.2 · p. 3
  • The singleton self-attention model generates blank tokens one by one, attending to known template tokens and tokens already generated.It ends a blank when it generates a special end-of-blank token, then moves to the next blank.Source: §3.2 · p. 3
  • Segment identity plus token offset localizes tokens because a single scalar position index cannot uniquely locate tokens when blank lengths are unknown.The model learns embeddings for these two-dimensional position indexes.Source: §3.2 · p. 3

4 Experiments

Experiments train models to recover masked text and compare them across random-mask and long-content settings; self-attention outperforms comparison methods in both settings.

4 Experiments

  • The experiments create supervised datasets by masking text portions with different strategies, then train models to recover the original text.Source: §4 · p. 3

4.1 Varying Mask Rates and #Blanks

  • As mask rate and blank count rise, BLEU and perplexity fall, while self-attention consistently outperforms seq2seq and GAN across settings and metrics.Table 1 covers mask rates of 30% and 50%.Source: Table 1 · p. 3, §4.1 · p. 3
  • On Yelp, masks varied by rate and blank count; ten annotators ranked generations on 40 test cases alongside BLEU and perplexity.The dataset contains 104K training and 1K test sentences.Source: §4.1 · p. 3
  • For a Yelp template with two blanks and 40% masking, self-attention produces a semantically coherent completion close to the gold text, unlike the other models’ ill-fitting patches.Source: §4.1 · p. 4, Table 2 · p. 4

4.2 Long Content Infilling

  • Long-content tests leave one noun and verb in Grimm templates (81.3% average masking) or player/team names and numbers in NBA templates (78.1%).Source: §4.2 · p. 4
  • On both long-content datasets, self-attention improves over comparison methods; automatic evaluation uses test-set perplexity because BLEU is less suitable for increasingly open-ended infilling.Source: §4.2 · p. 4, Table 3 · p. 4
  • In Grimm and NBA examples, self-attention tends to produce more reasonable, meaningful patches, though some output remains unsatisfactory.Source: §4.2 · p. 4, Table 4 · p. 4

5 Conclusion

The paper concludes that self-attention with global context modeling and segment-aware position embeddings improves over seq2seq and GAN-based models across supervised datasets. Its model uses a multi-head decoder, updates the template after each blank, and represents token positions by segment and within-segment offset.

5 Conclusion

  • Across a variety of supervised datasets, self-attention with global context modeling and segment-aware position embedding improved over seq2seq and GAN-based models.The authors present text infilling as a strong baseline for future research.Source: §5 · p. 4

A.1 Notations

  • The input is represented as segments, with notation for tokens within each segment, blank-boundary markers, filled segments, and the set of blank segment IDs.<bob> and <eob> mark each blank's boundaries; <bos> and <eos> mark the sentence boundaries.Source: §A.1 · p. 6

A More Details of Text Infilling Self-Attention Model

A.2.1 Template

  • After a blank is filled, its placeholder is replaced by the generated segment, and the decoder attends to the updated template for the next blank.During testing, the inserted segment is the inference output.Source: §A.2.1 · p. 6

A.2.2 Position Encoding

  • Token positions are identified by segment number and offset within that segment, allowing positional embeddings to distinguish locations in the template.The model uses sine and cosine functions of different frequencies, following the original Transformer.Source: §A.2.2 · p. 6

A.2.3 Applications of Attention

  • Decoder self-attention uses previous decoder outputs, while template-decoder attention uses template embeddings as memory to capture local and global information.Multi-head attention projects queries, keys, and values into different representation subspaces and combines their outputs.Source: §A.2.3 · p. 7

A.2.4 Training

Training fills each blank autoregressively using the template and generated tokens as context. The per-blank cross-entropy losses are summed, and Adam is used with a warmup and inverse-square-root learning-rate schedule.

A.2.4 Training

  • For each blank, the decoder predicts missing tokens autoregressively, conditioning on the template and what has already been filled in.The objective for the blank is cross-entropy.Source: §A.2.4 · p. 7
  • The loss for an infilling sentence is the sum of the cross-entropy losses for its blanks.Source: §A.2.4 · p. 7
  • Adam is used with β1 = 0.9, β2 = 0.997, and ǫ = 10−9; the learning rate warms up for 10,000 steps and then decays with the inverse square root of step number.The schedule uses const = 0.3.Source: §A.2.4 · p. 7

B Training Details

The appendix specifies how the Seq2Seq, self-attention, and GAN-based models are configured and used during infilling. The self-attention decoder receives the template as reference, while the GAN adds a discriminator to the Seq2Seq generator.

B.1 Model Parameters

  • The Seq2Seq model encodes the template and fills one blank at a time, using ground-truth blanks for decoder teacher forcing during training.Its appendix configuration uses a unidirectional LSTM encoder and a basic positional LSTM decoder, each with 1,600 units and one layer.Source: §B.1 · pp. 7–8
  • The GAN uses the Seq2Seq generator and trains a discriminator to distinguish generated infillings from ground truth for each blank.The discriminator’s classification result serves as a reward for updating the generator; the listed discriminator is a Conv1D classifier.Source: §B.1 · p. 8
  • The self-attention decoder uses the template as reference and receives a begin-of-blank token during inference.After filling a blank, the template is updated to guide infilling of the next blank.Source: §B.1 · p. 8

C Other Experiments

The experiments vary masking rates and blank counts, then examine longer infilling examples using anchoring words in Grimm Tales and NBA reports.

C.1 Varying Mask Rates and Segments

  • The varying-mask experiment evaluates 30%, 40%, and 50% token removal with templates containing one or two blanks; results are listed in Table 5.The supplied passage identifies the evaluation setup but does not provide the table’s outcome values.Source: §C.1 · p. 8

C.2 Longer Content Infilling

  • For longer Grimm Tales infilling, templates retain a noun and a verb as anchoring words, with two examples provided for comparison.Table 6 presents the examples.Source: §C.2 · p. 8
  • For NBA reports, templates retain a player or team name and a number-related phrase as anchors; Table 7 lists two examples.Source: §C.2 · p. 8

C.3 Preposition Infilling

The preposition-infilling experiment masks function words in Grimm Tales and compares generated examples. The self-attention model generates prepositions that fit the template, while Seq2Seq and GAN examples include grammatical mistakes.

C.3 Preposition Infilling

  • The dataset uses Grimm data with the same sentence count and vocabulary size as Section 4.2.Source: §C.3 · p. 8
  • Templates mask prepositions and articles, contain three blanks, and have an average mask rate of 20.9%.When fewer than three eligible spans exist, empty masks are added.Source: §C.3 · p. 8
  • The self-attention model generates prepositions that fit the template, whereas Seq2Seq and GAN examples show grammatical mistakes.Table 8 provides an example from the Grimm Tales preposition-masking data.Source: §C.3 · pp. 8–9, Table 8 · p. 10
Loading 1901.00158v2…