TESS: Text-to-Text Self-Conditioned Simplex Diffusion

1 Introduction

• Prior simplex diffusion is described as lacking self-conditioning and extensive downstream evaluation, while its 25-token generation blocks hinder infilling, flexible generation, and a global sequence view.Direct diffusion on the probability simplex avoids an extra mapping from diffused embeddings to discrete inputs or auxiliary methods such as binary encoding.Source: §1 · p. 1
• TESS performs diffusion over the entire sequence in vocabulary logit space, uses a fully non-autoregressive design, and introduces a new self-conditioning method.The paper reports that simplex-based self-conditioning substantially improves generation quality.Source: §1 · pp. 1–2
• TESS surpasses state-of-the-art non-autoregressive and diffusion-based approaches on NLG and is on par with a strong pretrained encoder-decoder model (Lewis et al., 2020).It also performs comparably to strong masked-language-model baselines on GLUE NLU tasks.Source: §1 · pp. 1–2
• TESS’s fully non-autoregressive approach is reported to sample faster and more efficiently than semi- and fully autoregressive methods for long sequences.The approach generates entire output sequences in parallel rather than using SSD-LM’s small semi-autoregressive blocks.Source: §1 · pp. 1–2

Training

• The forward diffusion process is a Markov chain that gradually adds Gaussian noise to data at each time step.The reverse process approximates the transition back toward the data distribution.Source: §2 · p. 2
• A transformer parameterizes the reverse process, and training minimizes mean squared error between the ground-truth data and its estimate.Sampling begins from standard Gaussian noise and gradually denoises it toward samples from the data distribution.Source: §2 · p. 2

Noise schedule

• The forward process uses the cosine schedule for α_t.Source: §2 · p. 2

Inference

• Inference iteratively denoises model predictions from time step T down to 1, starting from pure noise.Source: §2 · p. 2

Discrete diffusion for discrete domains

• Simplex-based diffusion applies diffusion to text without auxiliary methods for mapping categorical data into continuous space.The background attributes this procedure to Han et al. (2022).Source: §2 · p. 2

Continuous data representation and time-step embeddings

• TESS maps each token to a k-logit simplex, converts its probabilities into a weighted word-embedding vector, and adds a time-step embedding.The time-step embedding is added to the continuous input vector.Source: §3 · pp. 2–3

Text-to-text non-autoregressive modeling

• Unlike SSD-LM’s semi-autoregressive blocks, TESS processes the entire latent sequence in parallel and can generate sequences of any length.Evaluation tasks often require outputs of 100 tokens or more.Source: §3 · p. 3

Training

• TESS trains with cross-entropy between ground-truth tokens and predictions from a noisy logit simplex, rather than the unstable text-diffusion mean-squared-error objective.The passage describes that mean-squared-error objective as unstable for text diffusion models.Source: §3 · p. 3

Sampling

• At inference, TESS starts from Gaussian noise and reverses the diffusion process on the noisy k-logit simplex.The model’s prediction is treated as hypothetical ground truth and corrupted by the preceding number of diffusion steps to approximate a reverse step.Source: §3 · pp. 3–4

Self-conditioning

• Self-conditioning uses the model’s previous prediction alongside the current noisy input; TESS averages simplex probabilities before forming the continuous input.The probability-averaging approach is described as more efficient than concatenating and re-projecting the vectors, and is evaluated against that original method in Section 6.2.Source: §3 · p. 4

Variable sequence length

• Training pads variable-length outputs to a fixed length and includes padding tokens in cross-entropy; inference uses a specified maximum sequence length.Source: §3 · p. 4

4.1 Tasks and Datasets

• Evaluation covers QQP paraphrasing (147K positive pairs), NEWSELA-AUTO simplification (666K complex-simplified sentences), QUASAR-T question generation (119K document-question pairs), CNN-DailyMail summarization (300K articles and summaries), and GLUE classification.Source: §4.1 · p. 4

4.2 Baselines

• Baselines include GPT-2, BART, and GPVAE-T5, plus text-diffusion and iterative non-autoregressive models, including SSD-LM initialized from the same RoBERTa checkpoint as TESS.Results are reported without additional decoding methods such as minimum Bayes risk decoding.Source: §4.2 · p. 5

4.3 Evaluation

• Evaluation uses ROUGE variants for summarization; BLEU, ROUGE-L, and BERTScore for generation quality; D-1 and D-4 for diversity; and SARI for simplification.Source: §4.3 · p. 5

4.4 Implementation

• Implementation starts from pretrained RoBERTa, uses 1,000 inference steps for generation and 10 for classification, and trains with 5,000 steps at simplex scale k = 5.Source: §4.4 · p. 5

5.1 Paraphrase Generation

• TESS significantly outperforms GPT-2 and other non-autoregressive and diffusion baselines on paraphrase quality metrics while matching their diversity metrics.TESS also achieves competitive overall performance with BART, despite BART’s denoising pretraining and TESS’s lack of additional pretraining beyond RoBERTa’s masked-language-model checkpoint.Source: §5.1 · p. 5

5.2 Text Simplification

• TESS outperforms all baselines, typically by large margins, on NEWSELA-AUTO text simplification.The comparison includes autoregressive and non-autoregressive models.Source: §5.2 · p. 6

5.3 Question Generation

• TESS outperforms diffusion and non-autoregressive models on question-generation quality and diversity, and closely matches BART overall.Without pretrained RoBERTa initialization, it still outperforms all NAR baselines in BLEU and D-1 and remains close in BERTScore and ROUGE-L.Source: §5.3 · p. 6, Table 3 · p. 6

5.4 Summarization

• TESS achieves competitive results with BART on CNN-DailyMail and exceeds Diffuser by 1.9 ROUGE-L points and its bootstrapped variants by 0.8 points.It also outperforms GENIE while using half the number of diffusion steps.Source: §5.4 · p. 6, Table 4 · p. 6

5.5 Text Classification

• On GLUE, TESS matches or outperforms finetuned RoBERTa on several tasks, with roughly 2-point gains on MRPC and RTE.STS-B is recast as a 21-class classification problem.Source: §5.5 · p. 6

6.1 Variable Length Output

• TESS matches the target and BART output-length distributions, with generation quality consistent with BART across variable target lengths.For longer generations, TESS matches BART’s performance.Source: §6.1 · p. 6, Figure 2 · p. 7, Figure 3 · p. 8

6.2 Self-Conditioning

• The proposed self-conditioning variant delivers the best overall performance, and adding self-conditioning consistently improves results on text simplification and paraphrase generation.The ablation compares the proposed method with the original strategy and a model without self-conditioning.Source: §6.2 · p. 7, Table 6 · p. 7

6.3 Sampling Steps

• TESS performs well with relatively few sampling steps, with only a marginal performance drop when reducing from 1000 to 100; MRPC retains quality at 10 steps.The number of steps appears to correlate with task difficulty: MRPC requires few, while CNN-DM requires closer to 100 for good performance.Source: §6.3 · p. 7, Table 7 · p. 8

6.4 Sampling Speed

• TESS is substantially faster than SSD-LM and marginally faster than an equivalently sized BART when generating more than 2000 tokens.The speed comparison uses 100 diffusion steps; SSD-LM’s multiple blocks are especially costly when generating long sequences.Source: §6.4 · p. 7

6.5 TESS vs other Diffusion Methods

• TESS outperforms other diffusion methods across several benchmarks; the authors attribute this to its simplex formulation, simplex-based self-conditioning, and pretrained initialization.They note that TESS also outperforms other methods without pretraining, while suggesting that pretrained models can be used with relatively little extra training.Source: §6.5 · p. 7, Figure 3 · p. 8

• Continuous-domain diffusion work includes DDPMs, an SDE interpretation, sampling-step modifications, and classifier-free guidance for controllable generation.The cited applications include image, audio, video, and text-to-image generation.Source: §7 · p. 8
• Embedding-space diffusion methods include work limited to short sequences or small datasets, methods that cannot jointly train word embeddings, and sequence-to-sequence approaches.A related method was not evaluated on downstream NLP tasks; concurrent work explores noise manipulation and variational diffusion in embedding space.Source: §7 · p. 8
• Han et al. (2022) generate 25-token blocks left to right; TESS extends this approach to fully non-autoregressive generation and adds efficient simplex-aware self-conditioning.Han et al. (2023b) also extend simplex-based diffusion, focusing on models with more than 1B parameters.Source: §7 · pp. 8–9
• Discrete diffusion preserves discrete data structure, but these methods generally lag autoregressive models; TESS substantially outperforms Diffuser, which models generation as discrete edit operations.Earlier discrete diffusion work considered binary and categorical state spaces.Source: §7 · p. 9

• TESS combines fully non-autoregressive generation, vocabulary-logit-space diffusion, long-sequence support, and efficient self-conditioning.The conclusion presents it as a sequence-to-sequence model for language generation.Source: §8 · p. 9
• Across conditional generation and language-understanding tasks, TESS outperforms strong autoregressive baselines and recent text diffusion models while being more efficient than prior diffusion models.Source: §8 · p. 9
• Future work may pretrain TESS with denoising and infilling objectives, which the authors hypothesize could further improve performance.Source: §8 · p. 9

Limitations

• At fewer than 1,000 tokens, TESS remains slower than BART, while at 2,000 tokens it outperforms BART.The authors also report TESS quickly dominates semi-autoregressive generation.Source: Limitations · p. 9
• Ten sampling steps give promising results on most tasks but are not enough for strong performance on complex tasks such as summarization.The authors suggest that sampling-acceleration work in computer vision could further speed generation.Source: Limitations · p. 9
• RoBERTa’s absolute position embedding strategy limited the context window, constraining the scale used to assess TESS’s long-sequence potential.The authors primarily used RoBERTa-base models to facilitate fair comparisons with existing baselines.Source: Limitations · p. 9

Ethics Statement

• Language models are known to produce toxic and biased content (Sheng et al., 2021).The authors see little reason to expect TESS to be exempt from these issues.Source: Ethics Statement · p. 9
• The authors say TESS may also produce toxic and biased content, despite using an alternate modeling framework.They cite no evidence that the alternate framework removes these issues.Source: Ethics Statement · p. 9
• The authors hope future work will explore whether diffusion’s greater controllability can help reduce potential harms.They also call for examining how findings about harmful autoregressive generations transfer to this setting.Source: Ethics Statement · p. 9

A.1 Dataset

• The appendix lists NEWSELA-AUTO, QUASAR-T, and CNN-DailyMail for generation tasks, and the GLUE component datasets for NLU evaluation.Table 8 reports source and target sequence lengths for each dataset.Source: §A.1 · p. 13, Table 8 · p. 13

A.2 Baseline Details

• Baseline comparisons use reported results where available or where baseline code is unavailable, and TESS-tuned SSD-LM and BART receive the same tuning budget and decoding setting.GENIE is evaluated with decoding settings matched to TESS.Source: §A.2 · p. 13

A.3 Training Hyperparameters

• The appendix reports implementation and training details, including pretrained initialization, base-sized models, task-specific training steps, and roughly similar training times across models.For QQP on one GPU, TESS records 1.7 training steps per second, SSD-LM 1.8, and BART 1.4.Source: §A.3 · p. 13

A.4 Evaluation Package Details

• Evaluation uses sacrebleu for BLEU, rouge-score for ROUGE, mauve-text for Mauve, and bert-score for BERTScore; other metrics use the authors’ implementations.The authors say those other implementations will be open-sourced.Source: §A.4 · pp. 13–14

A.5 Inference Speed Experiments

• Inference-speed timings were measured on one 80 GB NVIDIA A100 using a 50-token prefix, 100 diffusion steps, and RoBERTa-large for all models.The reported values average five runs; Table 10 gives standard deviations.Source: §A.5 · p. 14, Table 10 · p. 15

B Sampling Steps

• TESS’s sampling-step ablation on question generation finds it remains largely unaffected by fewer steps and reaches parity with BART at 10 steps, while SSD-LM drops significantly.The authors caution that autoregressive and non-autoregressive forward-pass counts are not directly comparable; their comparison uses 74, the average number of BART tokens in question-generation responses.Source: B Sampling Steps · p. 14, Table 9 · p. 14

C Inference Step

• The appendix derives TESS’s inference update by substituting the signal-prediction variant into the DDPM step and using the simplex as the signal.The model prediction represents the ground-truth; the derivation uses a cosine schedule for ᾱ_t.Source: C Inference Step · p. 14

D Qualitative Examples

• TESS outputs natural CNN-DailyMail summaries that are often indistinguishable from BART samples in the authors’ qualitative comparison.Table 11 presents randomly chosen samples from both models.Source: D Qualitative Examples · p. 14, Table 11 · p. 15