PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion Glancing Transformer for Parallel Sequence to Sequence Learning

Lihua Qian, Mingxuan Wang, Yang Liu, Hao Zhou

arXiv:2212.10240v2cs.CL

TL;DR

Non-autoregressive models generate quickly but struggle to model multiple valid target modalities, limiting their generation quality. DIFFGLAT uses modality diffusion and residual glancing sampling to learn those modalities across decoding steps and layers. The paper reports improved accuracy and efficiency across sequence-generation tasks, including an average 0.47 BLEU margin over the Transformer with iterative decoding.

  • Problem

    Non-autoregressive models have difficulty modeling multimodal target distributions and have not consistently outperformed autoregressive baselines in generation quality.

    Source: §1 · p. 1

  • Method

    DIFFGLAT combines a discrete modality diffusion process that distributes modality learning across transitions with residual glancing sampling.

    Source: §3.1 · p. 4, §6 · p. 8

  • Results

    With iterative decoding, DIFFGLAT beats the Transformer by 0.47 BLEU on average and is reported to improve accuracy and efficiency across sequence-generation tasks.

    Source: §4.2 · p. 7, §6 · p. 8

  • Takeaways & Limitations

    The results demonstrate the potential of parallel generation to achieve superior accuracy and efficiency across multiple sequence-generation tasks.

    Source: §6 · p. 8

    The NAR factorization assumes conditional independence among target tokens.

    Source: §2 · p. 2

Abstract

from arXiv · show

Previously, non-autoregressive models were widely perceived as being superior in generation efficiency but inferior in generation quality due to the difficulties of modeling multiple target modalities. To enhance the multi-modality modeling ability, we propose the diffusion glancing transformer, which employs a modality diffusion process and residual glancing sampling. The modality diffusion process is a discrete process that interpolates the multi-modal distribution along the decoding steps, and the residual glancing sampling approach guides the model to continuously learn the remaining modalities across the layers. Experimental results on various machine translation and text generation benchmarks demonstrate that DIFFGLAT achieves better generation accuracy while maintaining fast decoding speed compared with both autoregressive and non-autoregressive models.

1 Introduction

Non-autoregressive generation offers parallel decoding and bidirectional context, but its difficulty modeling multimodal outputs has limited its ability to match autoregressive quality. DIFFGLAT combines modality diffusion with residual glancing sampling, and reports stronger generation quality while retaining fast decoding.

1 Introduction

  • DIFFGLAT is designed to outperform AR in both efficiency and accuracy without requiring knowledge distillation from AR.The model is designed within the denoising diffusion implicit model framework.Source: §1 · p. 1
  • A discrete modality diffusion process decomposes modality learning across transitions, while residual glancing sampling adjusts each transition’s learning difficulty layer by layer.Each transition includes a scheduled number of modalities, and adjacent NAR layers learn each transition.Source: §1 · p. 2
  • With 3 iterative decoding steps, DIFFGLAT beats the strong AR baseline by 0.47 BLEU; with 1 iteration, it outperforms all NAR baselines, including iterative ones.The paper also reports consistent comparisons with the AR Transformer across 10 machine-translation benchmarks in base and big settings.Source: §1 · p. 2
  • DIFFGLAT slightly outperforms the AR baseline on image-captioning and paraphrasing tasks, indicating potential beyond machine translation.Source: §1 · p. 2

2 Preliminary

The section introduces the conditional-independence factorization used by NAR models and diffusion processes that decompose sequence learning across steps. A synthetic proof of concept reports that progressive modality learning improves accuracy and captures additional modalities across layers.

2 Preliminary

  • NAR factorizes target generation with conditional independence, requiring the model to capture the modalities for all target-token combinations in one step.The section contrasts this with autoregressive prediction conditioned on preceding tokens and describes diffusion as interpolating intermediate distributions between source and target.Source: §2 · p. 2

2.1 Denoising Diffusion Models

  • Diffusion models define a generative process that fits the data step by step; non-Markovian processes are proposed for decomposing data modalities.The paper describes Markovian transitions as conditioning on the preceding state and motivates non-Markovian processes as less stochastic and more deterministic and efficient.Source: §2.1 · pp. 2–3

2.2 Proof of Concept

  • On synthetic data with four possible target modalities, progressive modality learning trains middle layers on two modalities and the last layer on all four.Source: §2.2 · p. 3
  • The vanilla NAR model fails to output correct sequences, while PML-trained middle and last layers gain accuracy; the middle layer learns two modalities and the last captures all modalities.The modality-distribution comparison is shown in Figure 2, and the synthetic results are presented in Table 2.Source: §2.2 · p. 3, Figure 2 · p. 3, Table 2 · p. 3

3 The Proposed Diffusion-GLAT

DIFFGLAT decomposes non-autoregressive modality learning into diffusion transitions and uses residual glancing to improve generation quality with few denoising transitions. Its generative process is self-conditioned, and its implementation supports different transition parameterizations.

3 The Proposed Diffusion-GLAT

  • DIFFGLAT aims to decompose modality learning to reduce non-autoregressive training difficulty while preserving high generation quality with few denoising transitions.It is presented in a denoising diffusion implicit model framework.Source: §3 · p. 3

3.1 Modality Diffusion Process: qMDP

  • qMDP interpolates target and model-prediction distributions across transitions, gradually adding modalities while preserving those already captured.The intermediate target has fewer modalities than the data target but more than the model prediction.Source: §3.1 · p. 4

3.2 Non-autoregressive Generative Process

  • The generative process conditions on model-computed inputs at every step, keeping training and inference consistent.The model propagates hidden states between transitions.Source: §3.2 · p. 4
  • Residual glancing samples tokens not captured by the model input, refining glancing layer by layer to guide learning of remaining modalities.The approach modifies token selection to focus on differences between the model output and the intermediate target.Source: §3.2 · p. 5

3.3 Implementation Details

  • For DA-Transformer training, the best alignment supplies the glancing target and qMDP input; inference extracts tokens using Joint-Viterbi decoding.Training also uses the highest-probability intermediate target and samples a diffusion timestep for each training step.Source: §3.3 · pp. 5–6
  • DIFFGLAT can use stacked layers for fully non-autoregressive models or share transition parameters for iterative models.A decoder forward pass can perform multiple denoising transitions by assigning part of its layers to each transition.Source: §3.3 · p. 6

4 Experiments

Experiments compare DIFFGLAT with autoregressive and non-autoregressive baselines across machine translation, paraphrasing, and image captioning. They report stronger generation results and analyze the diffusion process, residual glancing, and modality coverage.

4.1 Experimental Settings

  • Experiments cover machine translation, paraphrase generation, and image captioning, with models trained directly on raw data rather than AR-distilled data.The machine-translation experiments use 10 benchmarks.Source: §4 · p. 6, §4.1 · p. 6

4.2 Main Results

  • With three iterative decoding steps, DIFFGLAT beats Transformer by 0.47 BLEU on average; in one parallel iteration, it beats all previous NAR baselines and achieves 7.2× speedup at three iterations.The reported quality and speed results concern different decoding settings: one iteration for the NAR comparison, and three for the AR comparison and speedup.Source: §4.2 · p. 7
  • On 10 machine translation benchmarks, DIFFGLAT outperforms or matches the Transformer in both base and big settings.Table 4 reports BLEU scores for the 10 benchmarks.Source: §4.2 · p. 7, Table 4 · p. 7
  • For paraphrasing, DIFFGLAT beats the Transformer by about 1.7 BLEU and 2 ROUGE; for image captioning, it slightly beats the Transformer on four metrics.On image captioning, it also greatly improves over the DAT baseline.Source: §4.2 · p. 7, Table 5 · p. 7

Comparison of Different Diffusion Processes

4.4 Analysis

  • DIFFGLAT’s modality-coverage curves lie overall above DAT’s, and show greater coverage as the model performs more denoising steps.Coverage is estimated from training-data points whose normalized loss is at most a threshold; the exact real-world modalities are unavailable.Source: §4.4 · p. 8, Figure 7 · p. 8

5 Related Work

Related work covers non-iterative and iterative non-autoregressive generation methods, and situates DIFFGLAT among diffusion models for discrete data.

5 Related Work

6 Conclusion

DIFFGLAT combines modality diffusion and residual glancing sampling in a parallel sequence-generation model. The paper reports superior accuracy and efficiency to the autoregressive Transformer across multiple sequence-generation tasks.

6 Conclusion

  • DIFFGLAT is a parallel sequence-generation model trained with modality diffusion and residual glancing sampling.Source: §6 · p. 8
  • Smoothing modality learning in the diffusion framework significantly improves parallel-generation quality.Source: §6 · p. 8
  • Across multiple sequence-generation tasks, DIFFGLAT achieves superior accuracy and efficiency to the autoregressive Transformer.Source: §6 · p. 8

A Settings of the Synthetic Experiment

The synthetic experiment constructs source-target pairs with four known modalities and trains models to examine gradual modality learning. It evaluates token and sequence accuracy against the closest target.

A Settings of the Synthetic Experiment

  • The synthetic dataset has four modalities, with one randomly selected target modality per source sequence.It contains 100,000 training pairs and 5,000 pairs each for validation and test; source sequences have length 32.Source: A Settings of the Synthetic Experiment · p. 12
  • The modality-growing NAR model learns four modalities in its original loss and two merged modalities at the middle decoder layer.The merged targets pair modalities I with II and III with IV; the middle layer is trained to fit modalities II and IV.Source: A Settings of the Synthetic Experiment · p. 12
  • Number accuracy measures matching output tokens against the closest target, while sequence accuracy measures exact output-sequence matches.The experiment also assigns each output the modality of its closest target to report modality proportions.Source: A Settings of the Synthetic Experiment · p. 12

B Proof for Definition Consistency of the Modality Diffusion Process

The proof derives the modality-diffusion posterior using independent token distributions and establishes the stated result for t ≥ 1.

B Proof for Definition Consistency of the Modality Diffusion Process

C Additional Results on Machine Translation

DIFFGLAT improves over GLAT+CTC in the reported machine-translation results while retaining the original non-iterative inference process or allowing repeated decoder passes. In a separate latency comparison, three decoding iterations retain a 2.6× speedup over the Transformer (12-1).

C Additional Results on Machine Translation

  • 1∼3 BLEU points: DIFFGLAT improves over GLAT+CTC in the reported experiments.The authors state that DIFFGLAT can combine with existing parallel-generation methods while keeping its decoding process or adding iterations.Source: C Additional Results on Machine… · p. 13
  • DIFFGLAT keeps the original inference process for non-iterative decoding and can run the decoder multiple times without intermediate decoding for iterative inference.The reported CTC experiments are presented as a combination with DIFFGLAT.Source: C Additional Results on Machine… · p. 13

D Inference Time Comparison

  • 2.6× speedup: DIFFGLAT with 3 decoding iterations remains faster than the Transformer (12-1).Latency was measured on WMT14 with one Nvidia-V100 GPU and batch size 1.Source: D Inference Time Comparison · p. 13

E Hyper-parameters for Experiments

The experiments use task-specific dropout and model settings, and train first on the target before switching to modality diffusion. The appendix also specifies interpolation and residual-glancing schedules.

E Hyper-parameters for Experiments

  • For the first 100k training steps, models predict the target y0 at every step; subsequent training uses the modality diffusion process.Dropout is 0.1 for machine translation except En-Ro/Ro-En and Transformer big, which use 0.3; paraphrasing and image captioning use 0.3.Source: E Hyper-parameters for Experiments · p. 13
  • The simplified modality-diffusion implementation interpolates between P_{t−2} and y0, selecting γ_{t−1} as the maximum predefined-set value that preserves 90% of tokens in y0 ⊙ y_t.With this interpolation, y_{t−1} includes most target tokens correctly predicted in the previous y_t; the difference between y_t and y0 gradually reduces as t decreases.Source: E Hyper-parameters for Experiments · p. 13
  • Residual glancing uses μ = 1/T, while α decreases periodically from 0.5 to 0.1 for iterative DIFFGLAT.Source: E Hyper-parameters for Experiments · p. 13
  • Paraphrase models use 4 encoder and 4 decoder layers with hidden dimension 256; image captioning uses fast R-CNN features with a Transformer base setting.Both tasks are trained for up to 100k steps; DAT decoding length is 8 times source length for non-iterative models and 4 times for iterative models.Source: E Hyper-parameters for Experiments · p. 13

F Additional Analysis

The additional analyses examine decoding iterations, transition depth, diffusion length, evaluation metrics, and comparisons with related methods. They report where performance peaks and how DIFFGLAT differs in its use of intermediate targets and diffusion.

F Additional Analysis

  • BLEU rises as inference iterations increase to 3, then does not rise further; models trained with 3 iterations also score best at 3 inference iterations.The models were trained with generative processes of 3 iterations.Source: F Additional Analysis · pp. 13–14, Figure 8 · p. 14
  • 2 or 3 decoder layers per transition give the best performance, while 1 layer per transition lowers scores.The authors suggest the one-layer result may reflect insufficient modeling capacity; the total decoder-layer count and iterative decoding were held fixed.Source: F Additional Analysis · p. 14, Figure 9 · p. 14
  • On WMT14, performance grows until diffusion length T reaches 6 and declines at T = 8.Each reverse denoising step uses 3 decoder layers, so each iteration corresponds to 2 diffusion steps; the authors suggest larger iteration counts can make some transitions too easy to learn.Source: F Additional Analysis · p. 14, Figure 10 · p. 14

G The chrF and COMET Scores

  • DIFFGLAT has better chrF on WMT17 En↔Zh and comparable chrF on WMT14 En↔De than the Transformer, while its COMET scores exceed DAT but trail the Transformer.The authors suggest the COMET gap may arise from a distribution mismatch between NAR outputs and COMET training data.Source: G The chrF and COMET Scores · p. 15, Table 10 · p. 15, Table 11 · p. 15

H Data Statistics and Evalution Metrics

I Connections with Diffusion Models and Iterative NAR

  • Prior discrete-diffusion work includes uniform transitions, masking and similarity-based transition frequencies, token-information schedules, and route mechanisms.The cited discussion also surveys continuous diffusion approaches for discrete data.Source: I Connections with Diffusion Models… · p. 15
  • Unlike SUNDAE, which corrupts targets and uses them as denoising inputs, DIFFGLAT uses diffusion to divide modalities and treats corrupted sequences as intermediate training targets.The authors also report competitive quality without iterative decoding, whereas SUNDAE relies heavily on multiple decoding iterations.Source: I Connections with Diffusion Models… · p. 15
  • DIFFGLAT learns intermediate targets to capture multimodality gradually, whereas SMART trains mistake correction by replacing decoder inputs with generated outputs.Both methods can be used for iterative refinement.Source: I Connections with Diffusion Models… · p. 15

Related papers

Loading 2212.10240v2…