Source-linked AI summary
The Importance of Generation Order in Language Modeling
Nicolas Ford, Daniel Duckworth, Mohammad Norouzi, George E. Dahl
TL;DR
Neural language models usually generate left to right, leaving the effects of alternative token orders a question for investigation. This paper tests two-pass models that generate templates and fill their missing tokens, finding substantial quality differences across orderings, with function-first among the strongest strategies.
Problem
Neural language models commonly generate left to right, and the effect of alternative generation orders on model quality needs investigation.
Source: §1 · p. 1Method
The paper uses two-pass models that partition vocabulary, generate a partially filled template, and then generate the remaining tokens conditioned on it.
Source: §2 · pp. 1–2Results
Takeaways & Limitations
The experimental results justify further investigation of generation order in language and translation models.
Source: §1 · p. 1The authors lack sufficient evidence to decide whether function-first performance reflects syntactic structure or delaying commitment to rare tokens.
Source: §3 · p. 4
Abstract
from arXiv · showhide
Neural language models are a critical component of state-of-the-art systems for machine translation, summarization, audio transcription, and other tasks. These language models are almost universally autoregressive in nature, generating sentences one token at a time from left to right. This paper studies the influence of token generation order on model quality via a novel two-pass language model that produces partially-filled sentence "templates" and then fills in missing tokens. We compare various strategies for structuring these two passes and observe a surprisingly large variation in model quality. We find the most effective strategy generates function words in the first pass followed by content words in the second. We believe these experimental results justify a more extensive investigation of generation order for neural language models.
1 Introduction
Although neural language models commonly generate left to right, other generation orders are plausible; this paper isolates generation order and finds substantial differences among simple two-pass strategies. The strongest strategy it tried generates function words before content words.
1 Introduction
- Neural networks commonly generate text left to right, but multiple-pass generation and abstract representations are also plausible alternatives.The introduction notes that language may allow time for feedback between passes and cites freer word order in Czech and Polish.Source: §1 · p. 1
- The paper isolates generation order with a flexible, tractable architecture that permits exact sentence log-probabilities, finding substantial differences among simple two-pass strategies.The authors frame evidence of a large effect as needed before a broader search for improved generation strategies.Source: §1 · p. 1
- Function-first generation followed by content words performed best among the orderings tried, contrary to committing to a sentence topic before deciding its phrasing.The authors say these results justify further investigation of generation order for language and translation models.Source: §1 · p. 1
2 Two-pass Language Models
The paper constructs two-pass language models by partitioning vocabulary into first- and second-pass tokens, generating a partially filled template, then completing it. Because the template is deterministic, sentence probabilities can be computed exactly and partitioning strategies compared.
2 Two-pass Language Models
- The model partitions vocabulary into first- and second-pass tokens; the first pass retains its tokens in a same-length template and marks other positions with placeholders.The second-pass sequence contains the skipped tokens in their original order.Source: §2 · pp. 1–2
- A neural language model generates the template, then a conditional translation model generates the remaining tokens given that template.The fixed vocabulary partition gives a one-to-one correspondence between sentences and template/completion pairs.Source: §2 · p. 2
- A deterministic template makes sentence probabilities exactly computable, enabling exact log-likelihood learning and comparisons with other partitions and a single-pass model.The template is a deterministic function of the sentence, unlike latent generation orders that require prohibitive marginalization over permutations.Source: §2 · p. 2
- The implementation used two Transformer copies: one generated templates, and the other translated templates into complete sentences, with word embeddings shared across model components.The first copy had no encoder; the second was a sequence-to-sequence model.Source: §2 · p. 2
- The second pass predicts the complete target sentence but counts loss only at second-pass positions; the summed loss is divided by original sentence length.The paper calls this normalized loss the sentence's log-perplexity.Source: §2 · p. 2
- The tested partitions included frequency-based common-first and rare-first splits, function-first and content-first grammatical splits, and an odd-index control split.The function/content division was assigned using parsed LM1B training data; the odd-first split alternated frequency-sorted vocabulary indices between passes.Source: §2 · pp. 2–3
3 Experimental Results and Discussion
On LM1B, two-pass generation quality varied substantially with the vocabulary split: common-first and function-first performed best among the two-pass strategies, while the enhanced baseline slightly outperformed the best two-pass model. The authors suggest syntactic planning or delaying rare tokens as possible explanations, but lack evidence to choose between them.
3 Experimental Results and Discussion
- The experiments trained two-pass strategies and baselines on LM1B, using a 65,536-word lowercase vocabulary and a validation set drawn from one sixth of training data.The baseline comparison included both an ordinary Transformer and an enhanced baseline sized to match the two-pass models' parameter counts.Source: §3 · p. 3
- Common-first and function-first performed best among the two-pass strategies, while odd-first, rare-first, and content-first were worse than the baseline.Because several orderings were worse than the baseline, parameter-count increases alone do not explain the gains of common-first and function-first.Source: §3 · p. 3
- The enhanced baseline achieved slightly better perplexity than the best two-pass model, and larger Transformer embeddings appeared more effective than extra second-pass parameters.The two-pass model was intended to study generation order, not parameter efficiency.Source: §3 · p. 3
- One proposed explanation for function-first performance is that deciding syntactic structure first may make sentence generation easier.The authors also consider whether function-first results reflect overlap between function words and common words.Source: §3 · p. 3
- The authors lack sufficient evidence to choose between the syntactic-structure and delayed-rare-token explanations.They note that delaying a rare token may avoid conditioning later decisions on a low-probability event, especially when datasets cannot cover all utterances.Source: §3 · p. 4
4 Related Work
Related work explores non-left-to-right generation through multi-stage editing, syntax-based representations, and blank-filling approaches. Draft-editing models make perplexity intractable, whereas the paper's two-pass setup draws inspiration from blank filling and multi-scale PixelRNN.
4 Related Work
- Prior work reports that order affects model quality in machine translation, program synthesis, and text classification, while recurrent-network memory degrades over time.Attention is described as augmenting that memory.Source: §4 · p. 4
- Draft-editing models produce an initial decoding that is further edited, but their unrestricted drafts make perplexity intractable and require evaluation through external tasks.The passage contrasts this with the near-universal left-to-right generation of neural language models.Source: §4 · p. 4
- Prior syntax-based text models integrate explicitly designed or automatically learned parse structures into decoding.The cited examples include Chelba and Jelinek (1998), Yamada and Knight (2001), Graham and Genabith (2010), and Shen et al. (2018).Source: §4 · p. 4
- Blank-filling work by Fedus et al. (2018) resembles the model's second phase, and Multi-Scale PixelRNN by Van Oord et al. (2016) inspired its two-pass setup.Source: §4 · p. 4
5 Conclusion and Future Work
The paper concludes that its two-pass method reveals a strong effect of first-pass token choice on generation quality. It proposes parser-guided ordering and template completion for flexible-response systems as future directions.
5 Conclusion and Future Work
- The authors propose a two-pass procedure that generates selected tokens left to right, skips other positions, and then fills those positions; first-pass token choice strongly affected results.Source: §5 · p. 4
- Future work could use sentence parse trees to choose generation order and investigate which aspect of the function-first split was most helpful.Source: §5 · p. 4
- Filling templates with missing words may also support flexible responses in dialogue agents, question-answering systems, or systems combining hand-designed grammar with learned responses.Source: §5 · pp. 4–5