Source-linked AI summary
A Deep Generative Framework for Paraphrase Generation
Ankush Gupta, Arvind Agarwal, Prawaan Singh, Piyush Rai
TL;DR
The paper addresses automatic paraphrase generation, a problem important across NLP applications and constrained by the inadequacy of unconditional text-generation models for input-specific paraphrases. It combines LSTM sequence-to-sequence models with a VAE conditioned on the original sentence, and reports significant improvements on benchmark and question-paraphrase datasets.
Problem
Paraphrase generation matters across NLP applications, but traditional or unconditional generative models do not directly produce multiple paraphrases specific to a given input sentence.
Method
The framework combines LSTM sequence-to-sequence models with a VAE whose encoder and decoder are conditioned on the original sentence.
Results
The method outperforms state-of-the-art approaches by a significant margin on a general paraphrase dataset and demonstrates remarkable performance on a question paraphrase dataset.
Takeaways & Limitations
Conditioning both VAE sides on the input enables principled generation of multiple paraphrases for a sentence.
Abstract
from arXiv · showhide
Paraphrase generation is an important problem in NLP, especially in question answering, information retrieval, information extraction, conversation systems, to name a few. In this paper, we address the problem of generating paraphrases automatically. Our proposed method is based on a combination of deep generative models (VAE) with sequence-to-sequence models (LSTM) to generate paraphrases, given an input sentence. Traditional VAEs when combined with recurrent neural networks can generate free text but they are not suitable for paraphrase generation for a given sentence. We address this problem by conditioning the both, encoder and decoder sides of VAE, on the original sentence, so that it can generate the given sentence's paraphrases. Unlike most existing models, our model is simple, modular and can generate multiple paraphrases, for a given sentence. Quantitative evaluation of the proposed method on a benchmark paraphrase dataset demonstrates its efficacy, and its performance improvement over the state-of-the-art methods by a significant margin, whereas qualitative human evaluation indicate that the generated paraphrases are well-formed, grammatically correct, and are relevant to the input sentence. Furthermore, we evaluate our method on a newly released question paraphrase dataset, and establish a new baseline for future research.
Introduction
Paraphrase generation supports diverse NLP applications but has received relatively little prior work and has traditionally relied on rule-based methods. This paper introduces a conditional VAE–LSTM framework that generates multiple paraphrases for an input sentence and reports strong benchmark performance.
- Motivation: Paraphrase generation supports question answering, information retrieval, information extraction, summarization, and training-data creation for related learning tasks.In knowledge-based QA, paraphrasing can bridge user questions and knowledge-based assertions, while generated variants can improve recall or support other tasks.
- Background: Despite its importance, paraphrase generation has received relatively little prior work and has traditionally been addressed with rule-based approaches.Recent generative-model advances provide more powerful data-driven approaches to text generation.
- Contribution: The proposed framework combines LSTM sequence-to-sequence models with a VAE to generate paraphrases automatically from an input sentence.The architecture is presented as a novel, end-to-end deep learning framework.
- Model design: The VAE is conditioned on the original sentence through both encoder and decoder sides so generated text captures that sentence’s meaning.The input representation is obtained through an LSTM and used to condition both sides of the model.
- Novelty: Unlike existing sequence-to-sequence systems, the framework provides a generative mechanism for producing multiple semantically sensible paraphrases for each input sentence.Different latent-space samples produce alternative paraphrases rather than relying on lower-ranked beam-search variations.
- Evaluation: On benchmark datasets, the framework is reported to outperform state-of-the-art methods by significant margins and is evaluated quantitatively and qualitatively on question paraphrasing.The evaluation includes comparisons with sophisticated sequence-to-sequence models and the Quora questions dataset.
Methodology
The paper develops a VAE-LSTM architecture for generating paraphrases conditioned on an original sentence. Its encoder and decoder use LSTM representations, while latent sampling enables generation of multiple paraphrase variants.
- Architecture: The framework combines a variational autoencoder with LSTM sequence-to-sequence models in an end-to-end architecture for paraphrase generation.The VAE supplies the generative latent-variable component, while LSTMs learn sentence representations and decode paraphrases.
- VAE foundation: A VAE decoder maps random latent codes to observations, allowing the model to generate realistic text beyond previously seen encoded inputs.The latent code is distributed rather than deterministic, which supports generative sampling.
- Conditioning: Unlike a standard VAE, both encoder and decoder are conditioned on the original sentence to generate input-specific paraphrases.The decoder models pθ(x(p)|z, x(o)), incorporating the original sentence representation alongside the latent code.
- Training data and representations: The model trains on pairs of original and paraphrased sentences, whose vector representations are learned jointly by LSTM networks.The original sentence is converted to x(o), while the paraphrase representation x(p) parameterizes the VAE encoder; all LSTM parameters are learned end to end.
- LSTM components: The architecture contains three LSTM encoders and one LSTM decoder, totaling four LSTMs.One encoder represents the original sentence, another processes it with the paraphrase, and a third is used in the VAE input pathway.
- Decoder: The decoder receives the latent code z at every stage and the original-sentence vector x(o) for initialization when reconstructing the paraphrase.Both signals contribute to reconstructing the paraphrased sentence representation and output.
- Objective: Training maximizes a variational lower bound that balances expected paraphrase reconstruction with keeping the latent posterior close to the prior.The authors follow the training procedure used for VAE-based text generation, including the lower-bound objective.
Related Work
Prior paraphrase-generation research includes statistical machine translation and rule-based or data-driven approaches. Related generative text models use VAEs for sentence properties such as style, topic, and syntax.
- Translation-based approaches: Statistical machine translation systems have generated paraphrases from large sentence-pair collections and auxiliary paraphrase resources.These approaches include log-linear models combining paraphrase tables and feature functions.
- Generative text models: Other generative text models combine VAEs with controls or use them to model holistic sentence properties such as style, topic, and syntactic features.The paper situates its approach among controllable text generation and VAE-based sentence-generation work.
Experiments
Experiments evaluate the framework on MSCOCO and Quora using automatic metrics, qualitative examples, and human judgments. Results report improvements over baselines, strong performance for model variants, and a precision–recall trade-off under filtering.
- Datasets: The framework is evaluated on MSCOCO for standard paraphrase generation and Quora for question paraphrase generation.MSCOCO contains human-annotated image captions, while Quora provides paraphrased question pairs.
- Evaluation Metrics: Evaluation uses BLEU, METEOR, and TER, with higher BLEU/METEOR and lower TER indicating better performance.BLEU measures modified n-gram precision, METEOR incorporates stemming and synonyms, and TER counts edits needed to match a reference.
- MSCOCO Results: Both supervised MSCOCO variants outperform the state-of-the-art baseline, with VAE-SVG slightly better than VAE-SVG-eq.VAE-SVG achieves 4.7% absolute BLEU and 4% absolute METEOR improvement over the state-of-the-art.
- Quora Results: On Quora, VAE-SVG-eq performs best, and increasing training-data size improves results.Sharing the input-question encoder between the VAE encoding and decoding sides is identified as the primary reason for VAE-SVG-eq’s advantage.
- Quora Results: Compared with unsupervised VAE, the best variant gains more than 27% absolute BLEU and more than 19% absolute METEOR on Quora.Compared with VAE-S, the gains are almost 19% absolute BLEU and more than 10% absolute METEOR.
- Qualitative Evaluation: Human evaluation finds outputs close to ground truth in readability and relevance, while confidence filtering can raise BLEU by up to 55% at 10% recall.The filtering threshold improves precision while reducing recall; METEOR and TER show a similar trend.
Conclusion
The paper presents a VAE-based architecture augmented with sequence-to-sequence models that conditions both VAE sides on the input sentence to generate multiple paraphrases. It reports significant state-of-the-art improvements and strong performance on a question paraphrase dataset, with outputs that capture related new concepts.
- The proposed framework combines variational autoencoders with sequence-to-sequence models to generate paraphrases.
- Conditioning both encoder and decoder sides of the VAE on the input sentence enables principled generation of multiple paraphrases.
- The method outperforms state-of-the-art methods by a significant margin without hyper-parameter tuning.
- Evaluation on a recently released question paraphrase dataset demonstrates remarkable performance and establishes a baseline for future research.
- Generated paraphrases are semantically similar to the input while also capturing new concepts related to it.