Source-linked AI summary
Generating Sentences by Editing Prototypes
Kelvin Guu, Tatsunori B. Hashimoto, Yonatan Oren, Percy Liang
TL;DR
Existing sentence generators often generate from scratch, while the paper seeks a model that better represents diverse, complex utterances. It samples a training-corpus prototype and edits it with a neural editor and latent edit vector, improving perplexity and human-rated quality while exposing interpretable semantic structure.
Problem
Models that generate sentences from scratch can favor generic utterances, and diversity strategies can compromise grammaticality, motivating a model that better represents complex utterances.
Method
The prototype-then-edit model samples a latent training-corpus prototype and generates a sentence with a neural editor conditioned on a latent edit vector.
Results
The model improves perplexity, receives higher human-rated generation quality, and its edit vectors outperform standard sentence variational autoencoders on several semantic tasks.
Takeaways & Limitations
The model provides a semi-parametric sentence generator whose edit vectors reveal semantic structures such as similarity and sentence-level analogies.
Takeaways & Limitations
This paper focuses on unconditional generation; contextual inputs for tasks such as translation or dialogue are left as an extension.
Abstract
from arXiv · showhide
We propose a new generative model of sentences that first samples a prototype sentence from the training corpus and then edits it into a new sentence. Compared to traditional models that generate from scratch either left-to-right or by first sampling a latent sentence vector, our prototype-then-edit model improves perplexity on language modeling and generates higher quality outputs according to human evaluation. Furthermore, the model gives rise to a latent edit vector that captures interpretable semantics such as sentence similarity and sentence-level analogies.
1 Introduction
The paper proposes generating sentences by sampling a corpus prototype and editing it, addressing generic or less diverse outputs from models that generate from scratch. The model improves language-generation quality and supports semantic operations through its edit vectors.
- Current recurrent language models often favor generic utterances, while naive diversity strategies can compromise grammaticality.
- The prototype-then-edit model samples a training sentence, then uses a neural editor conditioned on a random edit vector to generate a new sentence.The editor attends to the prototype, which provides a grammatical and naturally diverse starting point.
- The model trains its neural editor by maximizing an approximation to the generative model’s log-likelihood.
- 13 points on Yelp and 7 points on the One Billion Word Benchmark are the model’s perplexity improvements over approaches that generate from scratch.Human evaluations also judge its generations higher quality.
- Latent edit vectors outperform standard sentence variational autoencoders on semantic similarity, locally-controlled text generation, and sentence analogy tasks.
2 Problem statement
The paper formulates sentence generation as selecting a latent prototype and applying a latent edit vector through a neural editor. It targets generation quality and interpretable, consistent semantic changes while focusing on the unconditional setting.
- The primary goal is to learn a generative sentence model for use as a language model.
- The process first samples a prototype sentence from the corpus and then samples an edit vector that conditions a neural editor generating a new sentence.
- The model is evaluated in the unconditional case, although contextual inputs could be supplied to the prototype selector and neural editor in applications such as translation or dialogue.
- Both the prototype x′ and edit vector z are latent variables in the model.
- The formulation exploits corpus sentences that can often be represented as minor transformations of other sentences; 70% of Yelp test sentences lie within word-token Jaccard distance 0.5 of a training sentence.
- The edit vector is intended to support semantic smoothness and consistent edit behavior across different sentences.
- Human evaluations report that the neural editor successfully captures both semantic properties.
3 Approach
The approach approximates prototype and edit-vector inference to make prototype-then-edit generation tractable. Lexical neighborhoods, an inverse neural editor, and variational training jointly support efficient learning and semantically meaningful edits.
- Approximate inference: The exact marginal likelihood is intractable because it sums over all prototypes and expects over an edit prior without closed form.The model therefore requires approximations for both latent prototypes and edit vectors.
- Approximate sum on prototypes, x′: The method restricts prototype summation to lexically similar sentences, using a Jaccard-distance neighborhood that can be precomputed with locality-sensitive hashing and minhashing.Assuming neighborhood size is constant, the resulting LEX(x) objective lower-bounds log p(x) up to constants.
- Lexical similarity semantics: 35.2% of sampled Yelp neighborhood pairs were exact paraphrases and 84% were at least roughly equivalent, supporting semantically meaningful edits.Only 7.2% were judged negated or changed in topic, and semantic similarity enables an edit model that prefers meaningful edits.
- Approximate expectation on edit vectors, z: Monte Carlo sampling from the edit prior has high variance because most sampled vectors yield near-zero likelihood, while a few important vectors matter substantially.The inverse neural editor q(z | x′, x) concentrates probability on edit vectors likely to map the prototype to the revised sentence.
- Approximate expectation on edit vectors, z: The expectation over edit vectors is lower-bounded with an ELBO, using q(z | x′, x) as a variational encoder and pedit(x | x′, z) as a decoder.The resulting objective combines the lexical prototype bound and ELBO, and is optimized by stochastic gradient ascent.
- Model architecture: The neural editor is an attention-based left-to-right sequence-to-sequence model that conditions its decoder on the prototype and edit vector.The edit prior samples vector length uniformly from 0 to 10 and direction uniformly on the unit sphere; the inverse editor infers vectors from edit pairs.
4 Experiments
The experiments evaluate NEURALEDITOR as a language model and as a model of sentence semantics. It improves perplexity and human-rated generation quality while supporting interpretable editing behavior and semantic tasks.
- Experimental setup: The evaluation covers language-modeling perplexity, human judgments of generation quality, and semantic properties of latent edit vectors.The semantic evaluations include similarity, locally controlled generation, and sentence analogy tasks.
- Language modeling: 13 points on Yelp and 7 points on the One Billion Word Benchmark: NEURALEDITOR improves perplexity over NLM.It surpasses the best NLM-and-KN5 ensemble on Yelp and nearly matches the ensemble on BILLIONWORD.
- Language modeling: NEURALEDITOR improves log-likelihood for a significant number of test sentences, especially when they are close to training-set prototypes.Figure 3 reports this proximity effect, unlike the N-gram comparison where NLM outperforms KN5 on most examples.
- Generation quality: 400 human evaluations find NEURALEDITOR's plausibility and grammaticality on par with the best temperature-tuned NLM, with higher diversity.Lowering NEURALEDITOR's temperature slightly improves grammaticality without substantially reducing diversity.
- Generation quality: Prototype-based editing preserves diversity at low decoder temperature, whereas a zero-temperature NLM collapses to one generic sentence.The prototypes provide diversity, while the decoder temperature controls how aggressively the model extrapolates from the training set.
- Semantics of NEURALEDITOR: NEURALEDITOR's edits are usually related paraphrases: only 1% are unrelated, and the model favors plausible syntactic transformations over random alternatives.It also generates longer, more diverse semantic walks than SVAE and sacrifices less semantic similarity at comparable attribute satisfaction.
5 Related work and discussion
The paper situates prototype-then-edit among attention-based, retrieval-augmented, semantically structured, and nonparametric approaches. Unlike conditional retrieval systems, it treats prototypes as latent variables and uses edit vectors to expose semantic structure.
- Attention and retrieval: Prototype-then-edit extends attention mechanisms beyond sequence-to-sequence models by attending to randomly sampled sentences.
- Attention and retrieval: Unlike prior retrieval models that condition on input context, the model addresses unconditional generation and marginalizes over latent prototypes.
- Attention and retrieval: Marginalizing prototypes makes attention depend on output-sequence similarity rather than input-sequence similarity.
- Nonparametric perspective: The approach resembles nonparametric kernel density estimation, with editing acting as learned smoothing noise around sampled training points.
- Semantic representations: The model uses a neural editor to generalize beyond remembered training sentences, with edit vectors revealing semantic structure.
6 Appendix
The appendix describes lexical-similarity graph construction and smoothing for NEURALEDITOR. Small smoothing probabilities preserve low perplexity because smoothing is needed mainly for a small subset of test sentences.
- Lexical-similarity construction: Locality-sensitive hashing maps sentences to lexical neighbors, forming a graph explored by breadth-first search from random seeds.
- Language-model smoothing: Small amounts of smoothing are sufficient for NEURALEDITOR to outperform the baseline NLM.
- Language-model smoothing: The model smooths its edit distribution with an NLM by introducing a special prototype ‘∅’ that can be edited into any sentence.
- Language-model smoothing: The smoothing equation linearly combines the edit model pedit with the NLM pNLM using the special-prototype probability p∅.
- Language-model smoothing: Smoothing is primarily needed to avoid degenerate loglikelihoods on a very small subset of the test set.