Source-linked AI summary
FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow
Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig, Eduard Hovy
TL;DR
Autoregressive decoding is sequential, while non-autoregressive models struggle with strong dependencies among output tokens. FlowSeq uses generative flow to model an expressive latent-variable prior, achieving comparable non-autoregressive translation performance with almost constant decoding time as sequence length grows.
Problem
Autoregressive decoding requires sequential generation, while naïve non-autoregressive models struggle to represent strong conditional dependencies among output tokens.
Method
FlowSeq uses generative flow to model an expressive conditional prior pθ(z|x) over sequential latent variables for non-autoregressive generation.
Results
FlowSeq achieves comparable performance with state-of-the-art non-autoregressive models on three translation benchmarks and almost constant decoding time with sequence length.
Takeaways & Limitations
FlowSeq combines modeling of the output joint distribution with efficient parallel decoding in a non-autoregressive seq2seq model.
Takeaways & Limitations
Iterative weight decoding does not require a separate pre-trained model but significantly slows decoding speed.
Abstract
from arXiv · showhide
Most sequence-to-sequence (seq2seq) models are autoregressive; they generate each token by conditioning on previously generated tokens. In contrast, non-autoregressive seq2seq models generate all tokens in one pass, which leads to increased efficiency through parallel processing on hardware such as GPUs. However, directly modeling the joint distribution of all tokens simultaneously is challenging, and even with increasingly complex model structures accuracy lags significantly behind autoregressive models. In this paper, we propose a simple, efficient, and effective model for non-autoregressive sequence generation using latent variable models. Specifically, we turn to generative flow, an elegant technique to model complex distributions using neural networks, and design several layers of flow tailored for modeling the conditional density of sequential latent variables. We evaluate this model on three neural machine translation (NMT) benchmark datasets, achieving comparable performance with state-of-the-art non-autoregressive NMT models and almost constant decoding time w.r.t the sequence length.
1 Introduction
Autoregressive seq2seq models support tractable token prediction but require sequential decoding, whereas non-autoregressive models decode in parallel yet struggle to model output dependencies. FlowSeq uses generative flow with latent variables to model these dependencies while retaining efficient decoding.
- Autoregressive generation: Autoregressive models factorize output probabilities by conditioning each token on previously generated tokens and the input.This enables tractable multi-class classification and maximum log-likelihood training, but requires a linear left-to-right generation pass.
- Non-autoregressive generation: Non-autoregressive models directly model the output-sequence joint distribution, decoupling decoding from generation history for parallel processing.A naïve formulation assumes target tokens are conditionally independent given the input.
- Modeling dependencies: Conditional dependencies among output tokens make simple non-autoregressive models perform far worse than autoregressive models.Latent variables can mitigate this problem by modeling those dependencies.
- Modeling dependencies: Fertility-based latent variables improve non-autoregressive performance but remain limited in expressing interdependence among target words.This limitation leaves performance behind state-of-the-art autoregressive models.
- FlowSeq: FlowSeq models an expressive prior distribution pθ(z|x) with generative flow to introduce more meaningful latent variables into non-autoregressive generation.The approach is presented as simple, effective, and efficient.
- FlowSeq: FlowSeq is a flow-based non-autoregressive seq2seq model that models the output joint distribution while enabling efficient parallel decoding.The paper describes it as the first non-autoregressive seq2seq model using generative flows, to the authors’ knowledge.
- Results: On WMT2014, WMT2016, and IWSLT-2014, FlowSeq achieves comparable performance with state-of-the-art non-autoregressive models and almost constant decoding time with sequence length.A typical left-to-right Transformer has super-linear decoding time with respect to sequence length.
2 Background
Flow-based generative models represent complex latent distributions by transforming simple priors through invertible mappings. Variational inference trains the latent-variable seq2seq model by optimizing an evidence lower bound when direct likelihood computation is intractable.
- Flow-based generative models: Flow-based models transform a simple distribution into a complex distribution through a chain of invertible transformations.This provides a framework for modeling complicated distributions such as the prior over latent variables.
- Flow-based generative models: A bijection between latent variables and a simple-prior space enables a generative process over the modeled variables.The inverse mapping is used to move between the two spaces.
- Flow-based generative models: The change-of-variable formula computes the complex density of z from the simple density of υ and the Jacobian determinant of the transformation.The Jacobian matrix is evaluated at z.
- Flow-based generative models: Efficient flows use transformations whose inverse functions and Jacobian determinants are tractable to compute.A stacked sequence of such transformations forms a normalizing flow.
- Variational inference and training: Maximum likelihood training minimizes negative log-likelihood, but marginalizing latent variables makes the likelihood intractable to compute or differentiate directly.This motivates variational inference with a parametric posterior qφ(z|y, x).
- Variational inference and training: The evidence lower bound combines reconstruction error log Pθ(y|z, x) with the KL-divergence between posterior and prior distributions.Both inference-model parameters φ and decoder parameters θ are optimized using this objective.
3 FlowSeq
FlowSeq combines variational latent-variable training with a generative-flow prior to produce target tokens independently and non-autoregressively. Its architecture uses invertible flow components and decoding approximations to make latent-variable generation practical.
- FlowSeq encodes the input, samples posterior latent codes during training, and optimizes the decoder and prior-flow probabilities through the ELBO.At test time, it samples latent codes from the prior flow before the decoder generates the target sequence.
- Generation of Latent Variables: The latent sequence z contains continuous vectors matching the target length, with each posterior variable modeled as a diagonal Gaussian.The Gaussian parameters are produced by neural networks such as RNNs or Transformers.
- Token Dropout: Token-level dropout discourages the posterior from encoding only each corresponding target token, encouraging z to capture contextual interdependence.Without this intervention, the model can reduce to the independent-token baseline.
- Decoder: The decoder processes the latent sequence without causal masking and predicts target tokens independently, preserving fully non-autoregressive generation.The architecture can use several layers of a neural sequence model such as a Transformer.
- Flow Architecture for Prior: The prior flow combines actnorm, invertible multi-head linear, and affine coupling layers in a multi-scale architecture for efficient density estimation.Splitting patterns model interactions across time or feature dimensions, while multi-head linear layers reduce determinant-computation cost.
- Decoding Process: Three approximate decoding algorithms reduce the otherwise intractable search over latent variables, while IWD avoids a separate pretrained model at the cost of slower decoding.The paper compares these decoding methods in Section 4.2.
4 Experiments
FlowSeq is evaluated against non-autoregressive and autoregressive baselines across three translation benchmarks, with analyses of decoding speed, rescoring, and translation diversity. It achieves strong translation quality while offering favorable batching and sequence-length scaling.
- Experimental Setup: FlowSeq is evaluated on WMT2014 DE-EN, WMT2016 RO-EN, and IWSLT2014 DE-EN benchmark datasets.The datasets contain approximately 4.5M, 610K, and 150K sentence pairs, respectively.
- Main Results: Table 1 compares FlowSeq argmax decoding with purely non-autoregressive baselines, separating models trained with and without knowledge distillation.The compared baselines include NAT w/ Fertility, NAT-IR, NAT-REG, LV NAR, CTC Loss, and CMLM.
- Main Results: More than 9 BLEU points: FlowSeq-base improves over baselines without knowledge distillation.The authors attribute this result to improved modeling of complex interdependencies in target languages.
- Main Results: Knowledge distillation benefits FlowSeq, but yields less than 3 BLEU improvement on WMT2014 DE-EN and no improvement on WMT2016 RO-EN.The authors hypothesize that FlowSeq’s stronger model is more robust against multi-modality.
- Main Results: FlowSeq achieves competitive performance against CMLM with 10 refinement iterations on WMT2014 and WMT2016, with only slight degradation in translation quality.The experiments did not use iterative refinement, although the authors leave combining refinement with FlowSeq for future work.
- Decoding Speed: 594% and 403% speedups: FlowSeq’s base and large models gain these improvements at batch size 128 relative to their base batch-size speeds.FlowSeq benefits more from increased batching than the autoregressive Transformer, whose incremental left-to-right search is less batching-friendly.
- Decoding Speed: As target sentence length increases, FlowSeq’s decoding time remains almost constant while the autoregressive Transformer’s time increases linearly.The relative speed advantage of FlowSeq therefore increases linearly with sequence length.
- Rescoring and Diversity: Rescoring improves consistently with more samples per length, whereas more length candidates do not necessarily improve performance; temperatures of 0.3–0.5 improve diversity and rescoring BLEU.At temperature 1.0, latent samples become noisy.
5 Conclusion
FlowSeq is proposed as an efficient and effective model for non-autoregressive sequence generation using generative flows. The authors identify iterative refinement and latent-space analysis as future directions.
- FlowSeq uses generative flows for efficient and effective non-autoregressive sequence generation.
- Future work could apply iterative refinement techniques, including masked language models, to improve translation quality.
- Future research could investigate FlowSeq’s latent space for deeper understanding and applications such as controllable text generation.
A Flow Layers
The flow-layer section includes a determinant-related term scaled by sequence length and hidden dimension, with the number of heads specified separately.
- The flow-layer expression contains T · h · log |det(W)|.
- Here, h denotes the number of heads.
B Model Details
This section presents a comparison of model size for the experiments.
- Table 3 compares model size across the experimental models.
C Analysis of training dynamics
Training dynamics show changing reconstruction loss during optimization, while FlowSeq avoids KL collapse because its decoder receives only the latent variable as input.
- Figure 7 plots training and development loss together with development BLEU scores over the first 50 epochs.
- Reconstruction loss initially increases, then decreases when training uses the full KL loss.
- FlowSeq does not suffer KL collapse because its non-autoregressive decoder has latent variable z as its only input.
D Analysis of Translation Results
FlowSeq generates multiple translation hypotheses that usually preserve the source meaning, while allowing varied phrasing through sampling controls. However, some outputs repeat or break because target-word dependencies are not directly modeled.
- Table 4 presents three hypotheses selected from 30 samples for each German input sentence on WMT14-DEEN.The examples use FlowSeq-base with l = 3, r = 10, and τ = 0.4.
- In most cases, FlowSeq accurately expresses the source meaning, sometimes differently from the reference in ways BLEU may not precisely reflect.
- Controlling length candidates, sampling temperature, and samples per length produces diverse translations expressing the same meaning.
- Repetition and broken translations still occur because dependencies between target words are not directly modeled.
E Results of Translation Diversity
Table 5 reports detailed translation-diversity results for FlowSeq-large on WMT14 EN-DE with knowledge distillation.
- Table 5 reports detailed results for translation diversity.
- The reported model is FlowSeq-large.
- The evaluation setting is WMT14 EN-DE with knowledge distillation.