Source-linked AI summary
Encode, Tag, Realize: High-Precision Text Editing
Eric Malmi, Sebastian Krause, Sascha Rothe, Daniil Mirylenka, Aliaksei Severyn
TL;DR
Text-generation tasks with high input-output overlap create a setting where standard seq2seq models use large vocabularies and substantial training data. LaserTagger instead predicts compact edit tags with BERT-based models, achieving comparable or better task performance, faster inference, and improved data efficiency, subject to limits on target reconstruction and word reordering.
Problem
Seq2seq and copy-based models can require large training sets and vocabularies even when target texts substantially overlap their inputs.
Method
LaserTagger reconstructs targets by applying edit operations to inputs and uses BERT-based tagging, including a BERT encoder with an autoregressive Transformer decoder.
Results
LaserTagger achieves state-of-the-art or comparable results on 3 out of 4 tasks, while its feed-forward tagger is up to 100x faster at inference.
Takeaways & Limitations
The approach is comparable on medium-to-large datasets, outperforms a strong seq2seq baseline with limited data, and can speed inference by more than two orders of magnitude.
Takeaways & Limitations
Arbitrary word reordering is not feasible, and targets requiring phrases outside the tag vocabulary are filtered from training data.
Abstract
from arXiv · showhide
We propose LaserTagger - a sequence tagging approach that casts text generation as a text editing task. Target texts are reconstructed from the inputs using three main edit operations: keeping a token, deleting it, and adding a phrase before the token. To predict the edit operations, we propose a novel model, which combines a BERT encoder with an autoregressive Transformer decoder. This approach is evaluated on English text on four tasks: sentence fusion, sentence splitting, abstractive summarization, and grammar correction. LaserTagger achieves new state-of-the-art results on three of these tasks, performs comparably to a set of strong seq2seq baselines with a large number of training examples, and outperforms them when the number of examples is limited. Furthermore, we show that at inference time tagging can be more than two orders of magnitude faster than comparable seq2seq models, making it more attractive for running in a live environment.
1 Introduction
LaserTagger treats high-overlap text generation as text editing, using sequence tags to reconstruct outputs from inputs. It achieves strong results across four tasks while reducing data and inference requirements.
- The approach uses a smaller output vocabulary and fixed output length, reducing training-data requirements relative to standard seq2seq approaches.
- The model encodes inputs, assigns edit tags to tokens, and realizes tagged tokens into output text.The three stages are Encode, Tag, and Realize.
- On four text-generation tasks, LaserTagger is comparable to seq2seq models with many examples and clearly better with fewer examples.
- LaserTagger frames text generation with overlapping inputs and outputs as a text-editing problem.
- LaserTagger combines BERT-based tagging with an autoregressive Transformer decoder, alongside a BERT-only tagger.
- LASERTAGGERAR achieves state-of-the-art or comparable results on 3 out of 4 tasks, while LASERTAGGERFF is up to 100x faster at inference.
2 Related Work
Related work includes edit-based and neural approaches for simplification, summarization, and grammatical error correction. LaserTagger builds on these lines while emphasizing compact, frequent phrase-based editing.
- Text Simplification: Text simplification systems use edit operations such as dropping, splitting, reordering, and lexical substitution.
- Text Simplification: Text simplification has also been addressed with phrase-based machine translation and neural encoder-decoder models.
- Text Simplification: A related text-editing model generates added tokens from the full vocabulary, unlike LaserTagger’s optimized frequent-phrase vocabulary.The full vocabulary may support more diverse output, while potentially affecting inference time, precision, and data efficiency.
- Summarization: Single-document summarization shortens texts while preserving meaning, using deletion-based methods or neural encoder-decoder models for abstractive edits.
- Summarization: The selected summarization dataset targets abstractive systems, contains short paragraphs, and covers multiple summarization operations.
- Grammatical Error Correction: Grammatical error correction detects and fixes learner errors using task-specific classifiers, unlabeled-data methods, and statistical machine translation.
3 Text Editing as a Tagging Problem
LASERTAGGER casts text editing as token-level tagging, using compact edit operations and a phrase vocabulary to reconstruct target texts. Training targets are aligned, converted into tags, and realized through rules, with special tags supporting task-specific transformations.
- Tagging operations: Each input token receives a KEEP or DELETE base tag, optionally paired with a phrase inserted before that token.The phrase belongs to a vocabulary V, and combined tags number approximately 2|V|.
- Tagging operations: Task-specific tags extend the basic operations, such as SWAP for reversing two source sentences in sentence fusion.SWAP applies to the first sentence’s final period and instructs realization to reorder the input sentences.
- Optimizing phrase vocabulary: The phrase vocabulary balances a small tag set against reconstructing as many target texts as possible.Selecting a vocabulary of at most ℓ phrases to maximize covered phrase sets is NP-hard.
- Optimizing phrase vocabulary: Candidate phrases come from target n-grams outside the source-target longest common subsequence, computed by dynamic programming in O(|s| × |t|) time.In practice, the vocabulary uses the ℓ most frequent phrases across phrase sets; frequent sentence-fusion phrases include discourse connectives.
- Converting training targets into tags: Once the vocabulary is fixed, targets are greedily converted into tags in O(|s| × n_p) time by matching source words and vocabulary phrases.Targets requiring phrases outside V are filtered out of the training data, although available phrases may still produce reasonable outputs.
- Realization: Realization applies rules that convert predicted tags into text, including capitalization handling and optional knowledge-base-dependent operations such as PRONOMINALIZE.Separating realization allows pronominalization only when the appropriate pronoun is confident and permits specialized rules for specific loss patterns.
4 Tagging Model Architecture
The tagging model combines a pretrained BERT encoder with a decoder that predicts edit labels. Its autoregressive variant models dependencies between tags and outperforms the feedforward alternative.
- Encoder: The tagger uses an encoder to produce activation vectors for input elements and a decoder to convert them into tag labels.The encoder is BERT-base with 12 self-attention layers, initialized from a pretrained case-sensitive checkpoint.
- Decoder: A feedforward decoder predicts each tag independently, whereas the proposed autoregressive decoder models dependencies between output labels.The autoregressive decoder is a single-layer Transformer consuming the previous predicted label embedding and encoder activations.
- Decoder: Directly consuming the encoder activation at the current step performed better and converged faster than full encoder-sequence attention in preliminary experiments.The authors attribute this to avoiding additional encoder-decoder attention weights.
- Decoder: LASERTAGGERAR, the autoregressive model, outperforms LASERTAGGERFF, the feedforward model.The paper uses these names for the autoregressive and feedforward variants, respectively.
5 Experiments
Experiments evaluate LASERTAGGER and strong BERT-based seq2seq baselines across four text-editing tasks. The tagging approach matches or exceeds baselines, is especially effective with limited data, and offers practical control and speed advantages.
- Experimental setup: LASERTAGGER is evaluated on sentence fusion, split and rephrase, abstractive summarization, and grammatical error correction.The experiments use strong Transformer baselines whose encoder and decoder replicate BERT-base, with pretrained encoder initialization.
- Sentence Fusion: A 500-phrase vocabulary reaches a plateau in Exact score and covers 85% of DfWiki training examples.The Gold curve treats the coverage as an upper bound for Exact score; smaller datasets may benefit from separately optimized vocabularies.
- Sentence Fusion: 2.7% Exact and 1.0% SARI gains over the previous 7-layer Transformer establish new sentence-fusion results for LASERTAGGERAR.SEQ2SEQBERT performs nearly as well, while omitting SWAP prevents reconstruction of 10.5% of the training set.
- Split and Rephrase: LASERTAGGER methods outperform the seq2seq baseline below circa 10k split-and-rephrase examples, with LASERTAGGERAR reaching 53.6% SARI from only 29 examples.At 29 examples, it predicts 5.2% of targets exactly correctly; on the full WikiSplit task, LASERTAGGERAR and SEQ2SEQBERT perform similarly and beat an earlier copying model.
- Abstractive Summarization: The tagger significantly outperforms summarization baselines, although text editing is not well-suited to complete paraphrases with zero lexical overlap.Low Exact scores are expected because many acceptable summaries exist; limited paraphrasing is sufficient for good empirical performance.
- Grammatical Error Correction: The tagging approach is more than seven times as accurate as the BERT-based seq2seq model for grammatical-error correction and is faster at inference.LASERTAGGERAR is already 10x faster than a comparably accurate seq2seq baseline at batch size 8; restricted flexibility also reduces certain generation errors.
6 Conclusions
LASERTAGGER frames high-overlap text generation as a smaller-vocabulary sequence-tagging problem, matching seq2seq performance on larger datasets while outperforming it with limited data and faster inference. Its scope is constrained by arbitrary reordering and morphologically richer languages.
- Conclusions: LASERTAGGER is a text-editing approach for high-overlap generation tasks that uses a smaller output tag vocabulary than typical seq2seq models.The approach formulates generation as sequence tagging rather than unrestricted text generation.
- Conclusions: LASERTAGGER has comparable performance on medium-to-large datasets and clearly outperforms a strong seq2seq baseline when training examples are limited.This pattern was demonstrated across four text-editing tasks.
- Conclusions: Tagging can speed inference by more than two orders of magnitude, making LASERTAGGER more attractive for production applications.
- Limitations: Arbitrary word reordering is not feasible, although deletion, insertion, and custom tags such as SWAP enable limited reordering.
- Limitations: Applying LASERTAGGER to morphologically richer languages may require a more sophisticated realizer to adjust word cases.
A Examples from Qualitative Analysis
The qualitative analysis contrasts characteristic seq2seq and LASERTAGGER errors. Seq2seq models produce imaginary words, premature endings, repetitions, and misleading hallucinations, while LASERTAGGER exhibits narrower but still consequential deletion and splitting errors.
- Imaginary words: Seq2seq models can generate nonexistent words by concatenating unrelated WordPieces, especially around rare input words.LASERTAGGER, trained at the word level, is immune to this specific error.
- Premature end-of-sentence: Seq2seq models may emit EOS prematurely, producing abrupt or even empty outputs, whereas this error is very unlikely for LASERTAGGER.
- Repeated phrases: Seq2seq models repeat words or phrases, while LASERTAGGER can only add words or phrases from its vocabulary.
- Repeated phrases: When sentence splitting lacks an obvious split, seq2seq models often repeat large fragments, whereas LASERTAGGER may leave the sentence unsplit or produce a lazy split.The tagger sometimes omits the splitting symbol or splits at a point that leaves an incomplete sentence.
- Hallucination: LASERTAGGER is less susceptible to hallucination, but inserted vocabulary items can still make outputs ungrammatical or odd.Seq2seq hallucinations are more likely to subtly misrepresent factual details while remaining fluent and credible.
- Misleading rephrasing through deletion: LASERTAGGER can alter sentence meaning through deletion, including removing negation and misrepresenting the input.The qualitative examples show deletion producing a factually reversed statement.
- Imaginary words: Seq2seq models can produce imaginary words in concrete outputs, such as corrupting a rare proper name during sentence splitting.