Improved Paraphrase Generation via Controllable Latent Diffusion

1 Introduction

• Earlier paraphrase approaches struggle to balance quality and diversity: deterministic models prioritize quality, while variational sampling can hinder complex-pattern quality.External semantic features can improve diversity, but are not always available.Source: §1 · p. 1
• Text diffusion’s rounding process adds decoding overhead and truncation errors, hindering efficiency and intervention.These issues motivate circumventing truncation for paraphrasing.Source: §1 · p. 2
• LDP achieves better and faster paraphrase generation than diffusion counterparts across various datasets.The paper also reports that LDP better balances generation quality and diversity than mainstream baselines.Source: §1 · p. 2
• LDP diffuses in a learned latent space bridged to text by an encoder-decoder, avoiding intermediate roundings and enabling source-segment semantic guidance without external features.The paper also reports viability for question generation and domain adaptation.Source: §1 · p. 2

2.1 Paraphrase Generation

• Paraphrase generation seeks diverse utterances that preserve meaning; deterministic models can limit diversity, external features may be inaccessible, and variational methods can sacrifice quality.External features include syntactic structures and exemplar syntax.Source: §2.1 · p. 2

2.2 Latent Diffusion Models with Control

• Diffusion reconstructs original inputs from Gaussian noise through a Markov noising process, then iteratively denoises toward valid data using a learned reconstructor.Sampling may use ancestral sampling, DDIM, or ODE solvers.Source: §2.2 · p. 2
• Latent diffusion applies diffusion to representations from a pretrained encoder-decoder, which prior work found more effective than diffusion on raw representations in several generation domains.The cited domains include video, audio, and image synthesis.Source: §2.2 · p. 2
• Plug-and-play control adapts a trainable copy of the diffusion model while freezing the learned model, injecting control signals through zero-convolution layers.The figure depicts LDP’s frozen encoder-decoder bridging text and latent diffusion, alongside a detailed denoiser architecture.Source: §2.2 · p. 3, Figure 1 · p. 3

3.1 Latent Diffusion Paraphraser

• LDP diffuses in a pretrained encoder-decoder latent space, bridging continuous generation and text decoding only at the beginning and end rather than rounding at every step.The encoder and decoder remain frozen during diffusion training, leaving the reconstruction network trainable.Source: §3.1 · p. 3
• The reconstruction network is trained on paraphrase pairs with source encoding as input, then samplers transform Gaussian noise into a source-conditioned latent representation for decoding.The model also uses sentence-level condition dropout during training; sequence length is determined by the end-of-sequence label.Source: §3.1 · pp. 3–4

3.2 Semantics Guidance by Controller

• The controller guides generation with semantic keyword segments, sampled from the longest 15% of input tokens, while a trainable replica of the base network is fused with the frozen model.At inference, keyword segments come from the source input; during fine-tuning, they come from reference paraphrases.Source: §3.2 · p. 4, Figure 2 · p. 4

4.1 Setups

• LDP uses 12 Transformer layers, BART-base, and a 768-dimensional diffusion space; sampling uses DPM-Solver++ with 25 steps.Training runs up to 250k steps, and the reported total model size is approximately 200M parameters.Source: §4.1 · p. 4
• The experiments use QQP, Twitter, and PAWS-wiki; QQP and Twitter each have 10k test and 10k validation sentences.Source: §4.1 · p. 5
• Baselines span Transformer and BART, variational and diffusion generators, and fine-tuned LLMs; evaluation uses BLEU, perplexity, div-4, and iBScore.iBScore combines BERTScore-based semantic similarity with a penalty for source duplication.Source: §4.1 · p. 5

4.2 Main Results

• LDP improves overall iBScore, especially div-4, over fine-tuned BART and outperforms SeqDiffuSeq on BLEU and perplexity.The authors attribute the diversity improvement to the diffusion module and describe LDP as a better bridge between diffusion and discrete text than rounding.Source: §4.2 · p. 6
• In human evaluation, LDP exceeds traditional and diffusion baselines, while LDP with semantic guidance achieves results comparable to SOTA LLMs.Source: Table 3 · p. 6
• Guiding generation with source-input segments improves paraphrase outputs without requiring external features.The examples use masked input segments such as “black money,” “no longer,” and “1000.”Source: §4.2 · p. 6, Table 2 · p. 6
• Across sampled paraphrases, LDP produces better and more diverse outputs than DiffuSeq and BART-CVAE, while DiffuSeq sometimes generates errors.Source: §4.2 · p. 6, Table 4 · p. 7
• LDP with semantic controller achieves the best iBScore among traditional and diffusion baselines, and LDP achieves the best performance among all baselines on QQP.The Table 1 caption also reports iBScore equivalent to open-source LLMs for LDP with semantic guidance, with higher BLEU scores.Source: Table 1 · p. 5, §4.2 · p. 6

5.1 Inference Efficiency

• LDP is 167.2 times faster than DiffuSeq and 50 times faster than SeqDiffuSeq; diffusion overhead is 18% of total timelapse.At 768 dimensions, LDP is still 10.4× faster than DiffuSeq in single-step text diffusion, excluding sampling acceleration.Source: Table 6 · p. 7, §5.1 · p. 7

5.2 Domain Adaptation by Controller

• Controller fine-tuning substantially improves ChatGPT-Aug test BLEU with only minor loss on the original test domain; without the controller, original-domain performance is retained.The evaluation uses a 10k-sentence test set and controller fine-tuning on ChatGPT-Aug data.Source: §5.2 · p. 7, Table 7 · p. 7

5.3 Ablation Study for Samplers

• With the same sampler setting, LDP outperforms DiffuSeq and SeqDiffuSeq, and it also outperforms them with fewer sampling steps.The comparison includes DDIM at 500 and 200 sampling steps.Source: §5.3 · p. 8, Table 8 · p. 8

5.4 Question Generation Ability

• On Quasar-T, LDP with enforced semantics achieves a BLEU score comparable to SOTA TESS, with much improved BERTScore.The evaluation uses BLEU and BERTScore on Quasar-T test data.Source: Table 9 · p. 8, §5.4 · p. 8

5.5 Diffusion Process during Generation

• During generation, BLEU increases as output length decreases; semantics shift markedly toward the target mid-process, followed by minor changes at smaller noise scales.Figure 3 tracks BLEU and averaged sentence length across diminishing noise scales.Source: §5.5 · p. 8, Figure 3 · p. 8

• Paraphrase generation is commonly modeled as end-to-end sequence generation, such as fine-tuned Transformers [31], but deterministic generation does not ensure diversity; researchers introduce external features or variational generation for diversity [62] [63] [23].Pretrained text encoders and decoders have also been introduced for generation [9].Source: §6 · p. 8
• DiffuSeq uses partial noising and rounding for end-to-end text generation [40], while BG-Diffuseq narrows the training-sampling gap with distance penalties and adaptive decay sampling [54].Source: §6 · p. 8
• Other approaches manipulate noise schedules to mitigate truncation [15], use encoder-decoder diffusion [41] [53], residual glancing [56], or logit-simplex diffusion [55].Source: §6 · p. 9

• LDP uses controllable latent diffusion for paraphrase generation and supports flexible semantic controls, including input segments rather than external features.Source: §7 · p. 9
• Experiments report better paraphrases with superior efficiency than diffusion counterparts, and a better quality-diversity balance than mainstream baselines.Source: §7 · p. 9
• Further analysis reports LDP’s versatility for related text-generation settings, including question generation and domain adaptation.Source: §7 · p. 9