Source-linked AI summary
PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification
Yinfei Yang, Yuan Zhang, Chris Tar, Jason Baldridge
TL;DR
Existing adversarial paraphrase research and challenging multilingual paraphrase datasets have focused mainly on English. PAWS-X addresses this gap with human-translated evaluation pairs across six languages and baseline experiments, finding that multilingual BERT and machine-translated training data perform best while the benchmark retains substantial headroom.
Problem
Most adversarial paraphrase research focuses on English, while existing multilingual datasets lack challenging examples needed to benchmark model improvements.
Method
PAWS-X extends PAWS evaluation and test examples with 23,659 human-translated paraphrase pairs in six languages and evaluates models with different capacities and multilingual training regimes.
Results
Multilingual BERT outperforms bag-of-words and ESIM, machine-translated data improves performance over zero-shot evaluation, and German, French, and Spanish perform better overall than Chinese, Japanese, and Korean.
Takeaways & Limitations
PAWS-X effectively measures sensitivity to word order, syntactic structure, and multilingual adaptability while providing a challenging benchmark for further multilingual paraphrase research.
Takeaways & Limitations
Translation introduces challenges such as inconsistent entity translations and translation noise that can cause incorrect predictions across languages.
Abstract
from arXiv · showhide
Most existing work on adversarial data generation focuses on English. For example, PAWS (Paraphrase Adversaries from Word Scrambling) consists of challenging English paraphrase identification pairs from Wikipedia and Quora. We remedy this gap with PAWS-X, a new dataset of 23,659 human translated PAWS evaluation pairs in six typologically distinct languages: French, Spanish, German, Chinese, Japanese, and Korean. We provide baseline numbers for three models with different capacity to capture non-local context and sentence structure, and using different multilingual training and evaluation regimes. Multilingual BERT fine-tuned on PAWS English plus machine-translated data performs the best, with a range of 83.1-90.8 accuracy across the non-English languages and an average accuracy gain of 23% over the next best model. PAWS-X shows the effectiveness of deep, multilingual pre-training while also leaving considerable headroom as a new challenge to drive multilingual research that better captures structure and contextual information.
1 Introduction
PAWS-X addresses the lack of challenging multilingual adversarial paraphrase data by extending PAWS to six languages. Experiments show that multilingual BERT and machine-translated training data substantially improve performance.
- 1 Introduction: Adversarial examples expose weaknesses in NLP models, while adversarial training data can improve performance for some models.
- 1 Introduction: PAWS-X extends adversarial paraphrase identification beyond English to Spanish, French, German, Chinese, Japanese, and Korean.The corpus contains 23,659 human-translated evaluation pairs.
- 1 Introduction: PAWS-X preserves high word overlap and semantic-similarity labels, probing models’ sensitivity to structure and context.
- 1 Introduction: Multilingual BERT improves accuracy by 32% over a bag-of-words model on PAWS-X.
- 1 Introduction: Machine translation outperforms zero-shot evaluation, while German, French, and Spanish perform better overall than Chinese, Japanese, and Korean.
2 PAWS-X Corpus
PAWS-X is constructed by translating PAWS examples into six languages, combining human-translated evaluation data with machine-translated training data. The resulting corpus supports multilingual transfer evaluation but is constrained by translation coverage and translation-specific challenges.
- 2 PAWS-X Corpus: PAWS-X translates the Wikipedia portion of PAWS from English into French, Spanish, German, Chinese, Japanese, and Korean.Human translators handle development and test sets, while neural machine translation handles the training set.
- 2 PAWS-X Corpus: Translation enables cross-lingual transfer tests because corresponding examples in every language originate from the same sources.The dataset can test whether sentences across languages are paraphrases.
- 2 PAWS-X Corpus: Evaluation translation uses professional native speakers, second-worker validation, and a final delivery target below 5% word-level error.
- 2 PAWS-X Corpus: Time and cost constraints prevented translating all 16,000 original development and test examples.
- 2 PAWS-X Corpus: Less than 2% of pairs remain untranslated on average and are excluded, often because of incompleteness or ambiguity.
- 2 PAWS-X Corpus: Translation introduces challenges such as inconsistent entity renderings and entity mentions appearing in only one sentence.
3 Evaluated Methods
PAWS-X evaluates models with different capacity to represent context and sentence structure, alongside multiple multilingual training and evaluation strategies.
- BOW uses unigram-to-bigram features and cosine similarity, classifying pairs as paraphrases above a cosine threshold of 0.5.
- ESIM encodes sentences with a BiLSTM and classifies their concatenated representations through a feed-forward layer.Its additional layers model more complex sentence interactions than cosine similarity.
- BERT is evaluated alongside simpler models as a model with greater capacity for sentence interaction and contextual representation.
- Translate Train machine-translates English training data into each target language, whereas Translate Test translates target-language test examples into English for evaluation.
- Multilingual BERT supports Zero Shot training on English followed by direct evaluation in other languages, and Merged training on English plus translated data in all languages.
4 Experiments and Results
Experiments compare multilingual models, training strategies, languages, and cross-lingual difficulty on PAWS-X. BERT substantially outperforms BOW and ESIM, while merged multilingual training performs best and performance is higher on Indo-European than CJK languages.
- Experimental Setup: Accuracy and AUC-PR are evaluated for BERT, BOW, and ESIM, with the best checkpoint selected using development-set accuracy.BERT uses positive-class probabilities for AUC-PR, whereas BOW and ESIM use a cosine threshold of 0.5.
- Model Comparisons: BERT outperforms BOW and ESIM by more than 15% absolute accuracy across all seven languages under both Translate Train and Translate Test.BERT Translate Train achieves an average 20% accuracy gain.
- Training/Evaluation Strategies: 8.6% accuracy and 7.1% AUC-PR average gains over Zero Shot are achieved when BERT trains on machine-translated examples in all languages.Zero Shot yields the lowest performance among the BERT strategies.
- Language Difference: Performance is consistently better on German, French, and Spanish than on Chinese, Japanese, and Korean across models and approaches.The difference is especially noticeable for Zero Shot, partly because machine translation works better for Indo-European languages and CJK languages differ more from English typologically and syntactically.
- Error Analysis: 61.7% of examples are correctly labeled in all languages by BERT-merged, while 32 examples fail in every language.Many examples failing across languages are hard or ambiguous, and some have incorrect gold labels or were incorrectly generated in the original PAWS data.
- Error Analysis: Some examples correctly predicted in only half of the languages fail because of translation noise, including inconsistent entity translations.Entity translation preferences can vary across translators according to their background knowledge.
5 Conclusion
The paper introduces PAWS-X, a challenging multilingual paraphrase-identification benchmark. Its results show sensitivity to word order and cross-lingual learning, while leaving headroom for further multilingual research.
- 5 Conclusion: PAWS-X contains 23,659 human-translated evaluation pairs in six languages.The dataset is presented as a challenging paraphrase-identification resource.
- 5 Conclusion: PAWS-X measures model sensitivity to word order and the efficacy of cross-lingual learning approaches.These properties are stated as findings from the experimental results.
- 5 Conclusion: PAWS-X leaves considerable headroom for multilingual research on paraphrase identification.The benchmark is intended to drive research that better captures structure and contextual information.