PaperlayerView exact revision on arXiv

Source-linked AI summary

Paraphrase Generation with Deep Reinforcement Learning

Zichao Li, Xin Jiang, Lifeng Shang, Hang Li

arXiv:1711.00279v3cs.CL

TL;DR

Paraphrase generation is useful but challenging, particularly because semantic similarity is difficult to evaluate reliably. The paper trains a Seq2Seq generator with a learned deep-matching evaluator and reinforcement learning, achieving better performance in automatic and human evaluations. A limitation is that learning the reward function is difficult when demonstrations are not optimal or the reward is not rigorously defined.

  • Problem

    Accurate and diverse paraphrase generation is difficult, and evaluation measures may undervalue semantically equivalent outputs with different wording.

  • Method

    RbM pairs a Seq2Seq generator with a deep matching evaluator, training the evaluator by supervised learning or inverse reinforcement learning and using it as a reinforcement-learning reward.

  • Results

    The proposed models outperform baseline and existing neural methods in automatic and human evaluations across Quora and Twitter paraphrase datasets.

  • Takeaways & Limitations

    A learned evaluator can provide accurate supervision that improves the generator’s paraphrase accuracy and quality.

  • Takeaways & Limitations

    Learning an optimal evaluator reward is difficult because expert demonstrations may be non-optimal and the reward function may lack a rigorous definition.

Abstract

from arXiv · show

Automatic generation of paraphrases from a given sentence is an important yet challenging task in natural language processing (NLP), and plays a key role in a number of applications such as question answering, search, and dialogue. In this paper, we present a deep reinforcement learning approach to paraphrase generation. Specifically, we propose a new framework for the task, which consists of a \textit{generator} and an \textit{evaluator}, both of which are learned from data. The generator, built as a sequence-to-sequence learning model, can produce paraphrases given a sentence. The evaluator, constructed as a deep matching model, can judge whether two sentences are paraphrases of each other. The generator is first trained by deep learning and then further fine-tuned by reinforcement learning in which the reward is given by the evaluator. For the learning of the evaluator, we propose two methods based on supervised learning and inverse reinforcement learning respectively, depending on the type of available training data. Empirical study shows that the learned evaluator can guide the generator to produce more accurate paraphrases. Experimental results demonstrate the proposed models (the generators) outperform the state-of-the-art methods in paraphrase generation in both automatic evaluation and human evaluation.

1 Introduction

Paraphrase generation is important for several NLP applications but remains difficult because accurate semantic equivalence is hard to evaluate. The paper proposes a learned generator–evaluator framework trained with deep learning and reinforcement learning.

  • Paraphrase generation creates different expressions that preserve a sentence’s meaning and supports applications including question answering, search, and dialogue.
  • Accurate and diverse automatic paraphrase generation remains challenging because natural language is complex.
  • Low-overlap paraphrases can receive very low ROUGE scores, exposing limitations of surface-form evaluation.
  • RbM combines a Seq2Seq generator with a deep matching evaluator, using evaluator feedback as reinforcement-learning rewards.
  • The evaluator can be trained with supervised learning or inverse reinforcement learning, depending on whether positive and negative examples are available.
  • Experiments on Quora and Twitter paraphrase datasets use automatic and human assessments, with the proposed models outperforming existing neural methods.

2 Models

The model defines paraphrase generation as Seq2Seq prediction and pairs a pointer-generator with a semantic sentence-matching evaluator. Attention, copying, positional information, and matching scores support generation and evaluation.

  • 2.1 Problem and Framework: Given input sequence X, the task generates output sequence Y with the same meaning as X.
  • 2.1 Problem and Framework: RbM uses a Seq2Seq generator with attention and copying, alongside a decomposable-attention evaluator that scores semantic similarity.
  • 2.2 Generator: Seq2Seq Model: The pointer-generator encodes inputs with an RNN and predicts each output word from decoder state, context, and the previous word.
  • 2.2 Generator: Seq2Seq Model: Attention computes context from encoder states, with α_ti representing attention weights and η the attention function.
  • 2.2 Generator: Seq2Seq Model: The pointer-generator mixes vocabulary generation with copying from the input, switching modes through a binary neural classifier.
  • 2.3 Evaluator: Deep Matching Model: The evaluator learns a real-valued matching function for sentence pairs using decomposable attention, word embeddings, inter-attended vectors, and positional encodings.

3 Learning

The paper learns a generator and evaluator jointly through deep reinforcement learning for paraphrase generation. The generator is pretrained with likelihood objectives, then optimized using evaluator-based rewards, while the evaluator is learned through supervised or inverse reinforcement learning depending on available data.

  • Generator learning: The generator is first trained with cross-entropy-based Seq2Seq learning, then fine-tuned with reinforcement learning using evaluator-provided rewards.Teacher forcing uses ground-truth previous words during likelihood training, creating exposure bias that motivates reinforcement-learning fine-tuning.
  • Generator learning: In the reinforcement-learning formulation, generated words are actions, preceding words define states, and the generator’s conditional probabilities form a stochastic policy.The evaluator supplies the terminal reward for each generated paraphrase, and policy gradient updates the generator.
  • Supervised evaluator learning: With positive and negative paraphrase examples, RbM-SL trains the evaluator as a classifier and then uses it as the generator’s reward function.The evaluator distinguishes positive paraphrase pairs from negative examples before reinforcement-learning updates are applied to the generator.
  • Inverse reinforcement learning: With positive examples alone, RbM-IRL learns a reward function from human paraphrase pairs using maximum-margin inverse reinforcement learning.The formulation addresses non-optimal expert demonstrations and an incompletely defined reward by imposing ranking properties on human and generated paraphrases.
  • Inverse reinforcement learning: RbM-IRL alternates evaluator and generator training, using generated sequences to make the evaluator more discriminative as outputs approach ground truth.After learning the evaluator, policy gradient improves the generator; the method can also use non-parallel sentences for subsequent generator reinforcement learning.
  • Training stabilization: Reward rescaling controls reward statistics and ranking information, stabilizing policy gradients under a changing evaluator; experiments report substantial gains for RbM-SL and RbM-IRL.Intermediate cumulative rewards are estimated by Monte Carlo simulation to provide denser supervision than a reward given only at sentence end.
  • Evaluator training: Curriculum learning samples simpler paraphrase pairs before more complicated ones by using edit distance as example difficulty.The evaluator first learns small modifications such as contractions, then gradually handles larger reformulations.

4 Experiment

Experiments on Quora and Twitter compare the proposed models with neural paraphrase-generation baselines using automatic and human evaluation. The proposed models generally outperform baselines, with evaluator-training methods and reinforcement-learning techniques affecting performance.

  • Experimental setup: The experiments compare RbM-SL and RbM-IRL with five neural paraphrase-generation baselines using automatic and manual evaluation.Baselines include attentive Seq2Seq, stacked Residual LSTM, VAE-SVG-eq, pointer-generator, and RL-ROUGE.
  • Datasets: The evaluation uses Quora question pairs and the Twitter URL paraphrasing corpus, with two Quora partitioning settings.Quora-I partitions by question pairs, whereas Quora-II prevents shared questions across training and test or validation data.
  • Automatic evaluation: RbM-SL and RbM-IRL outperform all baselines on every automatic measure in both Quora settings.On Quora, RbM-SL consistently exceeds RbM-IRL, and its evaluator reaches 87% accuracy on positive-versus-negative paraphrase identification.
  • Automatic evaluation: RbM-SL and RbM-IRL again outperform baselines on every automatic measure for Twitter, but RbM-IRL performs better than RbM-SL.The authors suggest that supervised evaluator training may be less effective on the relatively small Twitter dataset, while inverse reinforcement learning can leverage less data.
  • Training analysis: Reward rescaling particularly benefits the RbM models, while curriculum learning is necessary for RbM-IRL to achieve its best performance.Without curriculum learning, RbM-IRL performs comparably to ROUGE-RL; without reward rescaling, RbM-SL retains smaller gains over baselines.
  • Human evaluation: Human assessors rate the proposed models significantly higher than baselines for relevance and fluency, with RbM-SL best on relevance and RbM-IRL best on fluency.The evaluation samples 300 test sentences and uses paired t-tests with p-value < 0.01.

5 Related Work

Related work spans neural, bilingual, QA-specific, IRL, and GAN-based approaches to paraphrase generation. The paper distinguishes its evaluator-as-reward framework from GAN-style adversarial training and positions deep IRL as novel for Seq2Seq tasks.

  • Neural paraphrase generation: Neural paraphrase generation commonly formulates paraphrasing as sequence-to-sequence learning and explores capacity, vocabulary restriction, and diversity-oriented architectures.Examples include stacked residual LSTMs, additional vocabularies, and variational autoencoders.
  • Alternative settings: Bilingual approaches generate paraphrases through translation pivoting or back-translation, while QA-specific methods use external QA guidance and evaluation metrics.
  • Case study: Figure 3 presents generated-paraphrase examples from different models on Quora-II for comparison.
  • Inverse reinforcement learning: Inverse reinforcement learning learns reward functions from expert demonstrations, but the paper identifies its application of deep IRL to Seq2Seq as novel.
  • GAN and IRL: GANs pair generators with discriminators trained adversarially, whereas RbM-IRL uses an evaluator as an RL reward and maximizes expected cumulative reward.

6 Conclusion

The paper proposes a generator–evaluator framework for paraphrase generation, training the generator with Seq2Seq learning and reinforcement learning guided by a learned evaluator. Experiments show improved paraphrase quality over baseline methods, with future application to translation and dialogue.

  • The framework combines a Seq2Seq generator with a deep matching evaluator for paraphrase generation and identification.
  • The generator is first trained through Seq2Seq learning, while the evaluator is trained through supervised learning or inverse reinforcement learning.
  • A well-trained evaluator then supplies reinforcement-learning guidance that improves the accuracy of generated paraphrases.
  • Experiments demonstrate significantly better paraphrase-generation quality than baseline methods.
  • The authors plan to apply the framework and training techniques to machine translation and dialogue.

A Algorithms of RbM-SL and RbM-IRL

The algorithms train generators and evaluators iteratively using paraphrase, non-paraphrase, and non-parallel sentence corpora. RbM-SL trains the evaluator with labeled pairs, while RbM-IRL uses paraphrase pairs and non-parallel sentences.

  • RbM-SL: RbM-SL takes paraphrase pairs, non-paraphrase pairs, and non-parallel sentences as inputs, producing a trained generator.
  • RbM-SL: RbM-SL first trains the evaluator using positive and negative sentence pairs.
  • RbM-SL: The RbM-SL procedure repeatedly samples a sentence and generates a candidate output while iterating until convergence.
  • RbM-SL: RbM-SL estimates rewards through N Monte Carlo simulations and rescales the resulting reward before updating training.
  • RbM-IRL: RbM-IRL takes paraphrase pairs and non-parallel sentences and outputs trained generator and evaluator models.

11 end

The training procedure assigns a terminal evaluator reward to completed generated sentences and rescales rewards before optimization.

  • At the terminal step, the evaluator scores the generated sentence pair, while earlier steps use a continuation-based value expression.
  • The algorithm rescales the reward according to equation (8).

B Human Evaluation Guideline

The human evaluation guideline scores each paraphrase separately for grammar and fluency, and for coherence and consistency relative to the original question. Both dimensions use five-point scales, with examples clarifying how grammatical errors, information loss, and unknown tokens affect ratings.

  • Coherence and consistency: Coherence and consistency evaluates topical relatedness, preservation of question type, and informativeness without content loss.Judges assess helpfulness from the original poster’s perspective with respect to the original question.
  • Evaluation dimensions: Evaluators assign two separate scores from 1 to 5: one for grammar and fluency, and one for coherence and consistency.The criteria are intended to be judged independently rather than collapsed into a single rating.
  • Grammar and fluency: Grammar and fluency ranges from 5 for error-free text to 1 for non-sensical word compositions.Intermediate scores distinguish fluent text with minor or non-understanding grammatical errors from fully unacceptable language.
  • Unknown tokens: Unknown tokens must lower both scores below 5, with the coherence penalty determined by how much the token affects understanding.Examples assign coherence scores of 1, 3, and 4 when [UNK] causes serious loss, moderate loss, or little impact, respectively.
  • Scoring instructions: Grammar and coherence should be decoupled: a comprehensible but ungrammatical paraphrase can receive coherence 5 and grammar 3.A paraphrase that cannot be understood should instead receive low scores on both dimensions, even when it contains topic-relevant words.
Loading 1711.00279v3…