PaperlayerView exact revision on arXiv

Source-linked AI summary

PAWS: Paraphrase Adversaries from Word Scrambling

Yuan Zhang, Jason Baldridge, Luheng He

arXiv:1904.01130v1cs.CL

TL;DR

Existing paraphrase datasets provide too few high-overlap non-paraphrases, limiting evaluation of whether models understand word order. PAWS constructs such pairs through controlled word swapping and back translation, then validates them with human judgments. Models trained with PAWS data perform much better on the challenging dataset, while the results highlight the importance of non-local contextual information.

  • Problem

    Existing datasets contain too few high-overlap non-paraphrases, so they provide insufficient examples for evaluating sensitivity to word order.

  • Method

    PAWS generates high-overlap pairs using controlled word swapping and back translation, then retains clearly judged examples through human annotation.

  • Results

    Including 12k PAWS training pairs raises BERT’s PAWS accuracy from 33% to 85% without reducing QQP performance.

  • Takeaways & Limitations

    PAWS provides a resource for training and evaluating paraphrase identifiers on word order and syntactic structure.

  • Takeaways & Limitations

    Low-agreement pairs are excluded from experiments, although they remain in the released data for further study.

Abstract

from arXiv · show

Existing paraphrase identification datasets lack sentence pairs that have high lexical overlap without being paraphrases. Models trained on such data fail to distinguish pairs like flights from New York to Florida and flights from Florida to New York. This paper introduces PAWS (Paraphrase Adversaries from Word Scrambling), a new dataset with 108,463 well-formed paraphrase and non-paraphrase pairs with high lexical overlap. Challenging pairs are generated by controlled word swapping and back translation, followed by fluency and paraphrase judgments by human raters. State-of-the-art models trained on existing datasets have dismal performance on PAWS (<40% accuracy); however, including PAWS training data for these models improves their accuracy to 85% while maintaining performance on existing tasks. In contrast, models that do not capture non-local contextual information fail even with PAWS training examples. As such, PAWS provides an effective instrument for driving further progress on models that better exploit structure, context, and pairwise comparisons.

1 Introduction

High lexical overlap can conceal major meaning differences because word order and syntactic structure matter. PAWS addresses gaps in existing paraphrase data and shows that training on challenging examples improves model performance.

  • Motivation: High bag-of-words overlap does not guarantee paraphrase: reversing destinations can preserve words while changing meaning.The examples distinguish “Flights from New York to Florida” from “Flights from Florida to New York.”
  • Motivation: Existing QQP negatives rarely contain high-overlap non-paraphrases, leaving too few examples to teach models the importance of word order.Among approximately 1,000 QQP pairs with the same BOW, only 20% are non-paraphrases.
  • Contribution: PAWS introduces a workflow for generating high-overlap sentence pairs balanced between paraphrases and non-paraphrases.The dataset is constructed from Quora and Wikipedia sentences.
  • Results: 33% accuracy on PAWS is achieved by BERT fine-tuned on QQP, compared with over 90% accuracy on QQP.Including 12k PAWS training pairs raises PAWS accuracy to 85% without reducing QQP performance.
  • Results: PAWS training examples improve state-of-the-art models and expose whether models capture non-local contextual information.A simple BOW model fails to learn from PAWS examples, while gains correlate with model complexity.

2 Related Work

Prior paraphrase datasets mainly collect positive examples, while PAWS emphasizes challenging negative pairs and evaluates structural sensitivity directly. The dataset therefore supports research on representations that capture syntactic and non-local information.

  • Existing paraphrase data: Previous paraphrase datasets collected examples from sources such as image captions, tweets, subtitles, and back translation.These approaches focused primarily on collecting paraphrases.
  • Adversarial data: PAWS differs from prior work by emphasizing challenging negative examples with high lexical overlap.This targets cases where surface similarity conflicts with sentence meaning.
  • Adversarial data: Prior adversarial NLP methods mostly perturb inputs while preserving labels, using noise, word replacements, or syntactic transformations.Glockner et al. is identified as an exception that generated entailment and contradiction examples.
  • Structural evaluation: PAWS studies how well representations capture non-local contextual and structural information in paraphrase identification.Earlier structural approaches were evaluated on existing datasets, whereas PAWS emphasizes these properties directly.

3 PAWS Example Generation

PAWS combines controlled word swapping and back translation to generate well-formed, high-overlap pairs with different word order. Filtering and label-balancing steps produce diverse paraphrase and non-paraphrase examples.

  • Generation overview: A PAWS pair has high BOW overlap but different word order, and the generation method uses both swapping and back translation.The two strategies produce complementary label distributions and are combined for diversity and balance.
  • Word Swapping: Word swapping uses POS and named-entity templates, candidate sets from the input, and constrained beam search scored by a language model.Candidates are selected without replacement so the generated sentence preserves exactly the input’s BOW.
  • Word Swapping: The swapping method selects a best alternative sentence when its language-model score is within threshold t=3.0 of the input.The threshold balances generation quality and coverage, while heuristic rules prune 99% of simple list permutations.
  • Back Translation: Back translation uses English-German-English neural translation and filters outputs by BOW similarity and word-order inversion rate.German is used as the pivot because it produced more word-reordering variation with good translation quality.
  • Back Translation: BOW similarity is cosine similarity α between sentence word-count vectors; swapping pairs score α = 1.0, while back translation uses a relaxed threshold of 0.9.The relaxed threshold increases diversity and coverage while retaining high-quality paraphrases.
  • Back Translation: Inversion rate is the ratio of crossed word alignments, with the illustrated example yielding 9/15 = 0.6.Selected back-translation pairs include at least half with inversion rate above 0.02.
  • Label Balancing: Human annotations and recombination rules create a label-balanced set by combining mostly paraphrastic back translations with swapping examples.Pairs are added according to labels assigned to related source and generated sentences.

4 PAWS Dataset

PAWS is constructed from high-overlap sentence pairs generated through swapping and back translation, then filtered and labeled through human evaluation. The resulting datasets include labeled and auxiliary silver-labeled resources with documented splits.

  • Dataset construction: PAWS combines Quora and Wikipedia sentences into pairs with high bag-of-words overlap, word reordering, and both paraphrase and non-paraphrase labels.The datasets are named PAWSQQP and PAWSWiki.
  • Human filtering: 88% of generated examples passed human sentence-correction review on both domains.Accepted and fixed sentences proceeded to paraphrase annotation.
  • Human annotation: Five annotators gave binary paraphrase judgments, and only pairs with four or five agreeing raters were retained for the core experiments.Human agreement was 92.0% on Quora and 94.7% on Wikipedia before this final agreement filter.
  • Dataset splits: The final release contains 108,463 PAWS pairs across experimental splits, including PAWSQQP’s 12,665 pairs and the auxiliary PAWSWiki-Swap training set.PAWSWiki-Swap contains manually labeled swapped examples omitted from the final PAWSWiki dataset because they lacked back-translation counterparts.
  • Additional resources: An unlabeled PAWSWiki resource contains 656k pairs with silver labels assigned from the generation strategy.Swapped pairs are treated as non-paraphrases and back-translated pairs as paraphrases.

5 Evaluated Models

The evaluation covers models ranging from simple sentence encoders to architectures with increasingly rich contextual and cross-sentence interactions. Their representations differ in whether they model non-local context, word order, and word-by-word interactions.

  • Model coverage: The study evaluates six models spanning two baseline encoders and four advanced paraphrase-identification systems.The models are compared by their ability to represent non-local context and cross-sentential word interaction.
  • Baseline encoders: The BOW and BiLSTM baselines use cosine similarity between sentence encodings, with cosine values above .5 classified as paraphrases.The BOW encoder uses token unigram and bigram representations, while the BiLSTM produces contextualized sentence encodings.
  • Interaction models: ESIM encodes each sentence with a BiLSTM and uses additional multilayer processing to capture more complex sentence interaction than cosine similarity.Its pair representation concatenates the two sentence encodings before classification.
  • Interaction models: DecAtt computes word-pair interactions and aggregates aligned vectors but does not explicitly model word order, limiting its treatment of PAWS pairs.The paper identifies this modeling choice as a limitation on PAWS.
  • Interaction models: DIIN uses two-dimensional convolutions over n-gram pairs to extract high-order word-by-word interactions.It achieved state-of-the-art performance without relying on pretrained deep contextualized representations.
  • Contextual model: BERT pretrains a Transformer encoder on a corpus exceeding three billion words and fine-tunes it with one additional output layer.The paper describes BERT as a large pretrained contextual model.

6 Experiments

Experiments show that PAWS exposes failures in models trained on standard data, while PAWS examples substantially improve strong models without harming QQP performance. Results also link success to non-local context, training-set size, domain coverage, and model architecture.

  • Main Results on PAWSQQP: Under 40% accuracy is achieved by all models when trained on QQP and evaluated on PAWSQQP.All models exceed 83% accuracy on QQP, but transfer to PAWSQQP is poor.
  • Main Results on PAWSQQP: 27% is the BiLSTM–BERT accuracy gap on PAWSQQP, compared with 4.2% on QQP.The PAWSQQP comparison corresponds to a 60% relative reduction in error.
  • Main Results on PAWSQQP: 85% accuracy is reached by BERT after adding 12k PAWS training pairs, without reducing QQP performance.BERT retains 90.5% QQP accuracy when fine-tuned on QQP+PAWSQQP.
  • Size of Training Set: Both DIIN and BERT improve from 30% to 74% AUC with 6,000 PAWSQQP examples, and neither learning curve converges.The curves therefore indicate continued benefit from additional PAWS training examples.
  • Cross-domain Results: 70.5% cross-domain AUC is achieved by DIIN trained on Quora and tested on Wikipedia, versus 92.9% with in-domain training and 46.0% without PAWS.Training on both domains adds 9.2% absolute AUC on PAWSQQP over QQP+PAWSQQP alone.
  • Cross-domain Results: 12.1% AUC gain on PAWSQQP results from adding PAWSWiki to QQP training, while DIIN on PAWSWiki rises from 91.1% to 93.8%.The reported PAWSQQP values are 70.6% versus 58.5%.
  • BERT vs DIIN: BERT and DIIN make uncorrelated errors, with DIIN correcting 280 of 687 BERT mistakes on PAWSWiki.Their complementary errors suggest model combinations may improve performance.

7 Conclusion

PAWS joins diagnostic datasets by supplying challenging paraphrase pairs that expose sensitivity to word order and syntactic structure. The authors report that PAWS training improves state-of-the-art models on difficult examples and robustness to real-world examples.

  • 7 Conclusion: PAWS training data dramatically improves state-of-the-art models on challenging examples while making them more robust to real-world examples.The paper also presents PAWS as a resource for training and evaluating paraphrase identifiers.
  • 7 Conclusion: PAWS effectively measures model sensitivity to word order and syntactic structure.The conclusion frames this diagnostic role alongside other datasets targeting specific phenomena.
Loading 1904.01130v1…