Source-linked AI summary
Neural Paraphrase Generation with Stacked Residual LSTM Networks
Aaditya Prakash, Sadid A. Hasan, Kathy Lee, Vivek Datla, Ashequl Qadir, Joey Liu, Oladimeji Farri
TL;DR
Paraphrase generation lacks a prior deep-learning formulation despite its relevance to NLP applications. This paper explores sequence-to-sequence models and proposes stacked residual LSTMs, whose residual connections support deeper training and outperform other explored models across datasets and metrics.
Problem
Paraphrase generation is an important NLP subtask, but no prior work had proposed a deep learning-based formulation of it.
Method
The paper explores sequence-to-sequence models and introduces residual connections between stacked LSTM layers to train deeper LSTMs for paraphrase generation.
Results
The proposed model outperforms other models on BLEU and TER across all datasets, with broader advantages across Emb Greedy and METEOR depending on dataset and beam size.
Takeaways & Limitations
The experiments establish strong baselines for neural paraphrase generation across PPDB, WikiAnswers, and MSCOCO, including MSCOCO without using images.
Takeaways & Limitations
Paraphrase-specific evaluation metrics have limitations; PEM requires large in-domain bilingual corpora and human ratings and only models phrase-level paraphrasing.
Abstract
from arXiv · showhide
In this paper, we propose a novel neural approach for paraphrase generation. Conventional para- phrase generation methods either leverage hand-written rules and thesauri-based alignments, or use statistical machine learning principles. To the best of our knowledge, this work is the first to explore deep learning models for paraphrase generation. Our primary contribution is a stacked residual LSTM network, where we add residual connections between LSTM layers. This allows for efficient training of deep LSTMs. We evaluate our model and other state-of-the-art deep learning models on three different datasets: PPDB, WikiAnswers and MSCOCO. Evaluation results demonstrate that our model outperforms sequence to sequence, attention-based and bi- directional LSTM models on BLEU, METEOR, TER and an embedding-based sentence similarity metric.
1 Introduction
The paper focuses on neural paraphrase generation, a task with limited prior use of state-of-the-art deep neural networks. It explores sequence-to-sequence models and proposes residual connections for training deeper stacked LSTMs.
- Paraphrase generation produces a reference paraphrase from a source text and supports applications including question answering, information retrieval, summarization, and language generation.
- Traditional paraphrase-generation methods use hand-crafted rules or thesaurus- and corpus-based approaches.
- Sequence-to-sequence learning has achieved promising results across NLP tasks, but its application to paraphrase generation remained limited.
- The study evaluates multiple sequence-to-sequence models on three datasets using recognized automatic metrics.
- Residual connections between stacked LSTM layers are proposed to enable successful training of deeper LSTM networks.
2 Model Description
The model uses an encoder-decoder sequence-to-sequence framework with LSTM units and vertically stacked layers. Residual connections add earlier inputs to deeper representations, supporting the training of four-layer LSTM models without adding parameters.
- 2.1 Encoder-Decoder Model: The encoder converts a source sequence into a low-dimensional representation, and the decoder uses it to generate a target sequence.
- 2.1 Encoder-Decoder Model: Beam search evaluates a small set of candidate hypotheses to find a high-scoring decoded target sequence.
- 2.2 Deep LSTM: LSTM units combine the current input, previous hidden state, and previous memory state through input, forget, and output gates.
- 2.2 Deep LSTM: Vertical stacking passes the previous layer's hidden state to the next layer, producing recursively computed deep LSTM representations.
- 2.3 Stacked Residual LSTM: Residual connections are added after every two stacked LSTM layers as pointwise additions between compatible input and output dimensions.
- 2.3 Stacked Residual LSTM: The residual connection adds no learnable parameters and therefore does not increase model complexity like bidirectional models do.
3 Datasets
The study evaluates paraphrase generation across three datasets with substantially different characteristics: PPDB, WikiAnswers, and MSCOCO.
- Dataset comparison: The three datasets differ significantly in their characteristics, enabling evaluation of the approach across varied paraphrase-generation settings.The paper presents this diversity as evidence of the model’s versatility and robustness.
- PPDB: PPDB provides over 18M paraphrases, with short lexical, phrasal, and syntactic instances; the study omits syntactic examples and those containing numbers.The resulting data are suited to synonym generation and phrase substitution for lexical and phrasal paraphrasing.
- WikiAnswers: WikiAnswers contains approximately 18M word-aligned question pairs collected from user-tagged similar questions.The corpus can lose specialization between source and reference questions when users mark them as similar.
- MSCOCO: MSCOCO contains human-annotated captions for over 120K images, with five captions per image from different annotators.Although the captions are not guaranteed to be paraphrases, shared image content often leads annotators to describe the same prominent objects or actions.
4 Experimental Settings
The experiments compare four stacked-LSTM model types using controlled training and inference settings, including two beam sizes and a shared 512-unit layer width.
- Data splits: PPDB training used 90% of 5.3M paraphrases, while WikiAnswers used 4.8M training instances; each dataset had a 20K-pair test set.WikiAnswers vocabulary was clipped to the 50K most frequent words to reduce computational complexity.
- Models and inference: Four model types were evaluated with two- and four-layer stacked LSTMs, using beam sizes of 5 and 10 during inference.The bidirectional model used half the displayed layer count to maintain similar parameter sizes across models.
- Training: All models used one-hot word representations, SGD, ten training epochs, 50% dropout after each LSTM layer, and 512 LSTM units per layer.The learning rate started at 1.0 and was halved after every third epoch.
- Experimental scope: Training was limited to ten epochs and no hyper-parameter search was performed to allow exploration of many model variants.These restrictions define the reported experimental setting.
- Objective and diagnostics: Perplexity was used as the training loss, with lower values indicating better scores; beam search generated paraphrases at test time.WikiAnswers and MSCOCO showed strong training-validation correlation, whereas PPDB exhibited overfitting and worse validation perplexity.
5 Evaluation
The evaluation combines standard corpus metrics with embedding-based sentence similarity across three datasets, finding broad advantages for the residual LSTM while exposing metric and dataset effects.
- Metrics: BLEU, METEOR, and TER were used alongside an embedding-based sentence similarity metric to evaluate generated paraphrases.BLEU measures n-gram overlap, METEOR adds stemming and synonymy, TER counts edits, and the embedding metric compares phrases using word embeddings.
- Metric limitations: Existing metrics do not fully represent paraphrase meaning, novelty, or redundancy, motivating the additional embedding-based evaluation and indicating a remaining evaluation limitation.They may fail to reward entity paraphrases such as changing “London” to “Capital of UK”.
- Results: The proposed model outperformed other models on BLEU and TER for all three datasets.On METEOR, it performed best on MSCOCO and WikiAnswers, while sequence to sequence performed better on PPDB.
- Beam search analysis: For beam size 5, the proposed model outperformed the other models on all datasets; with beam size 10, attention was marginally better on Emb Greedy.The comparisons used single models without ensembles.
- Dataset and depth effects: Deeper LSTMs consistently improved performance over shallow models, while PPDB’s short phrases produced substantial training-validation differences.More than 50% of PPDB consists of one- or two-word phrases, which are penalized by metrics such as BLEU and METEOR.
6 Related Work
Prior paraphrase-generation work used knowledge-driven and statistical machine-translation approaches, while deep-learning applications remained limited. This paper situates residual recurrent connections among related deep-network developments.
- Knowledge-driven methods use hand-crafted rules or automatically learned paraphrase patterns, while other approaches rely on thesaurus-based or semantic-analysis resources.
- Figure 5 compares the distribution of sequence lengths, measured in words, across datasets.
- Phrase-based SMT frameworks learn paraphrases from aligned monolingual corpora, including news headlines or comparable news articles.
- Deep-learning applications for paraphrase generation had not been rigorously explored, motivating evaluation across several candidate datasets.
- Residual connections had been used in residual GRU image compression and DenseNet image recognition, but this work identifies recurrent neural networks as a new application.
7 Conclusion and Future Work
The paper presents stacked residual LSTMs for paraphrase generation and reports experiments across three large-scale datasets. It concludes that residual stacking is useful for this task while identifying further research directions and scope boundaries.
- The proposed technique trains stacked LSTM networks for paraphrase generation by extending sequence-to-sequence learning.
- Residual LSTM layers are reported as useful for paraphrase generation, but may not perform equally well for machine translation.
- Experiments on three large-scale datasets use various automatic evaluation metrics and include MSCOCO without leveraging its images.
- The experiments are intended to establish strong baselines that facilitate comparison and evaluation of subsequent paraphrase-generation work.
- Future work includes exploring learnable-memory networks, unsupervised deep learning, and image-captioning applications of neural paraphrase generation.