PaperlayerView exact revision on arXiv

Source-linked AI summary

Step-unrolled Denoising Autoencoders for Text Generation

Nikolay Savinov, Junyoung Chung, Mikolaj Binkowski, Erich Elsen, Aaron van den Oord

arXiv:2112.06749v3cs.CLcs.LG

TL;DR

Autoregressive text generation is slow to sample and poorly suited to arbitrary gap filling or iterative refinement. SUNDAE instead repeatedly denoises token sequences with training-time unrolling, achieving state-of-the-art non-AR WMT’14 EN→DE translation and good qualitative unconditional-generation results. Its non-autoregressive design also supports inpainting and collaborative text editing.

  • Problem

    Autoregressive models sample slowly and are poorly suited to filling arbitrary gaps or iteratively refining complete text drafts.

  • Method

    SUNDAE is a non-autoregressive denoising model that repeatedly samples transitions and feeds generated sequences back into its input, including unrolling during training.

  • Results

    SUNDAE achieves state-of-the-art results among non-AR methods on WMT’14 English-to-German translation and good qualitative results for unconditional text modeling on C4 and GitHub Python data.

  • Takeaways & Limitations

    SUNDAE’s non-autoregressive operation supports inpainting and new text-editing possibilities beyond left-to-right generation.

  • Takeaways & Limitations

    The model assumes sequences of fixed length N over a vocabulary of size v with a uniform prior p0.

Abstract

from arXiv · show

In this paper we propose a new generative model of text, Step-unrolled Denoising Autoencoder (SUNDAE), that does not rely on autoregressive models. Similarly to denoising diffusion techniques, SUNDAE is repeatedly applied on a sequence of tokens, starting from random inputs and improving them each time until convergence. We present a simple new improvement operator that converges in fewer iterations than diffusion methods, while qualitatively producing better samples on natural language datasets. SUNDAE achieves state-of-the-art results (among non-autoregressive methods) on the WMT'14 English-to-German translation task and good qualitative results on unconditional language modeling on the Colossal Cleaned Common Crawl dataset and a dataset of Python code from GitHub. The non-autoregressive nature of SUNDAE opens up possibilities beyond left-to-right prompted generation, by filling in arbitrary blank patterns in a template.

1 INTRODUCTION

SUNDAE addresses the slow, rigid, and architecturally constrained sampling of autoregressive text models with a non-autoregressive denoising model trained through unrolled transitions. It achieves state-of-the-art non-AR translation on WMT’14 EN→DE and strong qualitative results for unconditional generation and inpainting.

  • Autoregressive models sample slowly, require causal architectures, and handle gap filling or iterative refinement poorly.These limitations motivate non-autoregressive alternatives for more flexible text generation and editing.
  • Earlier non-autoregressive translation methods remained behind autoregressive benchmarks and often depended on distillation from larger autoregressive models.
  • The model repeatedly samples denoising transitions, feeding generated sequences back into the input during generation.Unrolled denoising also occurs during training, preparing the model for the noisy inputs encountered during sampling.
  • SUNDAE achieves state-of-the-art results on WMT’14 English-to-German translation among non-AR methods.
  • SUNDAE produces good qualitative unconditional-generation and inpainting results on C4 and GitHub Python data.
  • Training unrolls are crucial for the proposed method’s performance, according to the paper’s ablation analysis.

2 METHOD

SUNDAE models text generation as a time-homogeneous Markov chain that repeatedly denoises token sequences, with training designed to expose the model to states encountered during generation. Sampling begins from random or prompted sequences and uses improved decoding strategies to accelerate convergence.

  • 2 METHOD: The model assumes a uniform prior over fixed-length vocabulary sequences and uses a conditionally factorized transition distribution whose later chain distributions can become more expressive.Although the one-step distribution has product structure, subsequent transitions are not restricted in the same way.
  • 2.1 TRAINING WITH UNROLLED DENOISING: Unrolled denoising trains on short chains beginning from corrupted data so the model learns to denoise states likely to occur during full sampling.This avoids directly optimizing the full chain, whose likelihood is intractable and whose discrete sampling limits gradient flow.
  • 2.1 TRAINING WITH UNROLLED DENOISING: The training objectives use reconstruction losses after one or more denoising steps, with logits providing the quantities optimized by the neural transition model.The paper identifies the one-step and unrolled objectives as logits loss and unrolled logits loss.
  • 2 METHOD: SUNDAE samples sequentially from a learned transition function, starting from a random sequence or prompt and feeding each output back into the chain.The model uses a fixed number of Markov steps during generation.
  • 2.2 SAMPLING: Low-temperature sampling generates high-quality samples in 10-16 steps, while argmax-unrolled decoding and updating fewer tokens provide additional convergence-speed strategies.Updating a random subset of tokens is used when diversity is especially important, such as unconditional generation.
  • 2.2 SAMPLING: Table 1 compares AR and non-AR systems on WMT’14 EN→DE and DE→EN using test BLEU, including candidate reranking and inference-step information.The table highlights the best non-AR systems and distinguishes SUNDAE from systems using AR reranking.

3 EXPERIMENTS

SUNDAE is evaluated on translation, unconditional generation, and inpainting, combining non-autoregressive decoding with iterative denoising. It achieves strong raw non-AR translation results and qualitatively coherent, bidirectional text completion.

  • 3.1 MACHINE TRANSLATION: SUNDAE uses WMT’14 EN↔DE and EN→FR corpora, evaluates BLEU, and removes decoder causality from a Transformer encoder-decoder.The experiments compare raw non-AR models separately from AR-distilled systems.
  • 3.1 MACHINE TRANSLATION: Translations begin from random token arrays and are refined for multiple iterations using either low-temperature sampling or faster argmax-unrolled decoding.Argmax-unrolled decoding resamples low-certainty tokens using unrolled logits, while low-temperature sampling remains stochastic.
  • 3.1 MACHINE TRANSLATION: 26.25 BLEU on EN→DE made SUNDAE the best raw non-AR model and brought it within 1.05 of Transformer base without AR assistance.At T = 10, low-temperature sampling also exceeded Mask-Predict and Easy-First by 1.5 BLEU.
  • 3.1 MACHINE TRANSLATION: 26.57 BLEU with L(1:2), versus 11.19 with L(1), shows that at least one unrolled denoising step is crucial for translation performance.Adding a third loss term did not improve the score, reaching 26.25 BLEU.
  • 3.2 TEXT GENERATION: Inpainting experiments show SUNDAE using bidirectional context to fill arbitrary masked spans in C4 text and GitHub Python code.The model can transfer information across separated context regions and complete code that requires jointly interpreting surrounding variables and functions.

4 RELATED WORK

Prior work established strong autoregressive text generation but motivated non-autoregressive alternatives for faster decoding and broader conditioning. Non-autoregressive research spans diffusion, denoising, and translation methods, though translation systems often trail autoregressive quality or depend on distillation.

  • Non-autoregressive translation methods such as CMLM and DisCo offer excellent decoding speed but remain behind autoregressive benchmarks and rely on autoregressive distillation for competitive performance.
  • Diffusion-based non-autoregressive models recently showed promise for unconditional language modeling, addressing limitations of deterministic decoding under multimodality.
  • Non-autoregressive generation has been explored through latent variables, normalizing flows, denoising autoencoders, and iterative translation systems.
  • Denoising objectives became central to text representation learning through BERT and RoBERTa, while this work differs by using random-token corruption and unrolled denoising.

5 CONCLUSION

The paper presents SUNDAE as a non-autoregressive denoising model whose training unrolls the denoising process. It reports strong non-autoregressive translation results, qualitative unconditional modeling, and inpainting capabilities.

  • SUNDAE achieves state-of-the-art WMT’14 English-to-German translation among non-autoregressive methods without distillation from large autoregressive models.
  • SUNDAE also produces good qualitative unconditional text on C4 and demonstrates inpainting on C4 and GitHub Python data.

A METHOD DETAILS

SUNDAE trains and samples a denoising Markov chain, using corruption, unrolled denoising, target-length prediction, and temperature-controlled sampling. The method supports iterative translation refinement and reaches convergence in fewer steps than autoregressive decoding in the illustrated example.

  • A METHOD DETAILS: Target-length prediction is trained with cross-entropy, teacher-forced with ground-truth lengths, and supplied with the most likely predicted length during sampling.
  • A METHOD DETAILS: Corruption randomly replaces a sampled proportion of token positions with vocabulary tokens, producing a multinomial-diffusion-like forward process with per-token independence except for the corruption proportion.
  • A METHOD DETAILS: The target-length module predicts downsampled lengths, using a factor of 2 so the maximum prediction outcome is 64 when the maximum target length is 128.
  • A METHOD DETAILS: The illustrated German-to-English process converges after 3 steps, compared with 10 autoregressive steps for a 10-token translation.

E TRANSLATION ANALYSIS

Translation quality improves through iterative sampling: additional steps correct repeated-token errors, while temperature controls the speed–diversity trade-off. Very low temperatures improve scores quickly but can eventually underperform higher temperatures.

  • E TRANSLATION ANALYSIS: Repeated-token multimodality errors are corrected after a few sampling steps because each step conditions on the previous one.
  • E TRANSLATION ANALYSIS: Very low temperatures initially improve model scores faster but can be overtaken by higher temperatures, while very high temperatures improve slowly.

F DISTILLATION SCORES OBTAINED WITH TRANSFORMER–BASE

This section compares SUNDAE with Imputer using data distilled from an autoregressive Transformer-Base model on both translation directions. SUNDAE outperforms Imputer at 8 steps for EN→DE and DE→EN.

  • SUNDAE noticeably outperforms Imputer at 8 steps on both EN→DE and DE→EN translation.The comparison uses models trained on data distilled from an autoregressive Transformer-Base model.
  • Table 6 reports test BLEU scores for SUNDAE and Imputer under the distilled-data setting.
  • The evaluation covers English-to-German and German-to-English translation tasks.

G SACREBLEU SCORES FOR WMT’14 EXPERIMENTS

This section reports WMT’14 translation scores without autoregressive distillation. SacreBLEU is used to compute and report the BLEU measurements.

  • SacreBLEU computes the reported BLEU scores without requiring manual tokenization of references and candidate translations.
  • The SacreBLEU-based scores are presented in Table 7, with BLEU⋆ denoting scores computed using SacreBLEU.
  • The WMT’14 experiments report test BLEU scores without autoregressive distillation.

H UNCONDITIONAL SAMPLES FOR C4

This section presents unconditional samples from SUNDAE trained on the C4 dataset. The samples generally resemble reasonable-quality internet text, with newline symbols common in both the web-crawled data and outputs.

  • SUNDAE’s C4 unconditional samples generally resemble reasonable-quality internet texts.Sample #4 is identified as a possible exception.
  • The C4 samples are presented without cherry-picking.
  • Newline symbols are abundant in both C4 training data and the generated samples because C4 was crawled from the web.

I UNCONDITIONAL SAMPLES FOR EMNLP2017 NEWS

This section presents unconditional SUNDAE samples trained on EMNLP2017 News and gives pseudocode for training and sampling. The samples are evaluated without cherry-picking and compared with ScratchGAN outputs.

  • I UNCONDITIONAL SAMPLES FOR EMNLP2017 NEWS: SUNDAE’s EMNLP2017 News samples are presented without cherry-picking and alongside ScratchGAN samples for comparison.
  • I UNCONDITIONAL SAMPLES FOR EMNLP2017 NEWS: The News samples use temperature 0.8, and none appear in the training set.The passage presents this as evidence that the model does not merely memorize its training data.
  • I UNCONDITIONAL SAMPLES FOR EMNLP2017 NEWS: SUNDAE’s pseudocode separates training through build_loss_fn from sampling through sampling_fn.
  • I UNCONDITIONAL SAMPLES FOR EMNLP2017 NEWS: The pseudocode’s corruption step mixes random text with the original batched text using a mask.
  • I UNCONDITIONAL SAMPLES FOR EMNLP2017 NEWS: Sampling repeatedly applies the model for a specified number of steps, while training supports unrolled denoising steps.

K LONGER UNCONDITIONAL SAMPLES FOR C4

This section presents selected unconditional samples from SUNDAE trained on 256-token C4 sequences and compares them with D3PM samples from a different 128-token LM1B setup.

  • SUNDAE samples are drawn from a model trained on longer 256-token sequences from the C4 dataset.Table 10 contains selected unconditional samples from this model.
  • Because the datasets and experimental setups differ, the comparison is intended only to provide hints rather than a controlled evaluation.
  • D3PM samples provide a comparison based on 128-token sequences from the LM1B dataset.The comparison uses samples from a recent discrete diffusion paper.
Loading 2112.06749v3…