Source-linked AI summary
A Continuously Growing Dataset of Sentential Paraphrases
Wuwei Lan, Siyu Qiu, Hua He, Wei Xu
TL;DR
Paraphrase research lacks large parallel corpora with meaningful coverage and negatives. The paper links tweets sharing URLs to collect and annotate sentential paraphrases, producing a large corpus and continuously growing resource. It reports 51,524 labeled pairs and more than 30,000 monthly paraphrases at approximately 70% precision.
Problem
Existing paraphrase corpora are limited by scale, meaningful negative examples, lexical skew, or manual topic selection.
Method
The paper collects sentential paraphrases from tweets referring to the same URL and filters redundant manual retweets before annotation.
Results
51,524 sentence pairs form the largest human-labeled corpus to date, while more than 30,000 new paraphrases are captured monthly at approximately 70% precision.
Takeaways & Limitations
The method provides a continuously growing paraphrase resource and supports up-to-date phrasal paraphrase extraction for downstream NLP tasks.
Takeaways & Limitations
PIT-2015 requires manual topic selection because automatically detected topics may be incorrect or too broad.
Abstract
from arXiv · showhide
A major challenge in paraphrase research is the lack of parallel corpora. In this paper, we present a new method to collect large-scale sentential paraphrases from Twitter by linking tweets through shared URLs. The main advantage of our method is its simplicity, as it gets rid of the classifier or human in the loop needed to select data before annotation and subsequent application of paraphrase identification algorithms in the previous work. We present the largest human-labeled paraphrase corpus to date of 51,524 sentence pairs and the first cross-domain benchmarking for automatic paraphrase identification. In addition, we show that more than 30,000 new sentential paraphrases can be easily and continuously captured every month at ~70% precision, and demonstrate their utility for downstream NLP tasks through phrasal paraphrase extraction. We make our code and data freely available.
1 Introduction
Paraphrase research lacks large, parallel corpora with meaningful negative examples. This paper introduces a Twitter-based collection method and a continuously growing resource for downstream NLP.
- Paraphrases support applications including question answering, semantic parsing, information extraction, machine translation, textual entailment, and semantic similarity.
- 51,524 sentence pairs form the paper’s new gold-standard corpus, described as the largest to date.
- Tweets referring to the same URL provide the basis for collecting sentential paraphrases without relying on prior classifier or human selection.
- The Twitter News URL Corpus includes meaningful non-paraphrases and multiple references, supporting paraphrase identification and paraphrase-generation evaluation.
- More than 30,000 new sentential paraphrases can be automatically labeled each month at approximately 70% precision.
- The method enables up-to-date phrasal paraphrase extraction from Twitter for downstream use.
2 Existing Paraphrase Corpora and Their Limitations
Existing publicly available corpora are limited in scale, negative-example quality, or collection efficiency. The MSRP corpus is skewed toward lexical overlap, while PIT-2015 requires manual topic selection.
- Two publicly available corpora combine sentential paraphrases and non-paraphrases: MSRP and PIT-2015.
- MSRP contains 5,801 sentence pairs from clustered news articles, with 4,076 for training and 1,725 for testing.
- MSRP is skewed toward over-identification and contains many sentence pairs sharing numerous ngrams.
- PIT-2015 contains 13,063 training pairs on 400 topics and 972 test pairs on 20 topics, with many lexically divergent paraphrases.
- PIT-2015 requires manual topic selection because automatically detected topics can be incorrect or too broad for focused annotation and classification.
3 Constructing the Twitter URL Paraphrase Corpus
The corpus links tweets sharing URLs to collect paraphrase candidates, filters manual retweets, and combines automated harvesting with human annotation. This process yields a large, quality-controlled corpus and supports continuous collection of new paraphrases.
- 3 Constructing the Twitter URL Paraphrase Corpus: URL tracking collects large quantities of candidate paraphrase pairs without automatic news clustering or topic detection.The method retrieves tweets containing the same resolved URL, using news-account data as its source.
- 3.2 Filtering of Retweets: Manual retweet filtering removes redundant tweets that differ only through copying, punctuation, or linked-page metadata.Filtering uses substring, punctuation, and HTML-tag rules after tokenization and sentence splitting.
- 3.2 Filtering of Retweets: After filtering, higher PINC and lower Jaccard scores indicate more substantially rephrased tweets.PINC measures ngram-based dissimilarity, while Jaccard measures token-based string similarity.
- 3.3 Gold Standard Corpus: Human annotation presents an original sentence with 10 candidates to six annotators, then assigns pair labels by majority vote.Annotators select candidates with the same meaning; each question took about 53 seconds on average.
- 3.3 Gold Standard Corpus: The gold-standard corpus contains 42,200 training tweets from 4,272 URLs and 9,324 test tweets from 915 URLs.The authors describe it as the largest gold-standard paraphrase corpus to date.
- 3.3 Gold Standard Corpus: 300 expert-annotated pairs produced Cohen’s kappa of 0.739 against the majority vote, with worker-vote precision 0.871, recall 0.787, and F1 0.827.Worker quality was also monitored using agreement with other workers, and higher-quality workers labeled more data.
- 3.4 Continuous Harvesting of Sentential Paraphrases: The raw-tweet method continuously produces more than 30,000 new sentential paraphrases per month at about 70% precision.This automatically generated collection is described as a silver-standard paraphrase corpus.
4 Comparison of Paraphrase Corpora
The three corpora contain distinct, complementary paraphrase phenomena, and their differing lexical characteristics affect which identification models perform well. Surface-overlap, semantic, and Twitter-specific models therefore show dataset-dependent strengths.
- 4.1 Paraphrase Phenomena: The three datasets exhibit distinct and complementary compositions of paraphrase phenomena.MSRP has more synonyms, PIT-2015 more phrasal paraphrases, and the URL dataset more elaboration, spelling, and anaphora phenomena.
- 4.2.2 Model Performance and Dataset Difference: Nearly 80% of MSRP sentence pairs are paraphrases, an atypical distribution that may cause falsely positive predictions.This random baseline reflects the corpus's skew toward paraphrases.
- 4.1 Paraphrase Phenomena: MSRP contains more ngram overlap, PIT-2015 shorter and more lexically dissimilar sentences, and the URL corpus falls between them while resembling PIT-2015.These differences are measured with the PINC metric, which is opposite to BLEU.
- 4.2.1 Models: The LR model using lemmatization and ngram-overlap features achieves very competitive performance across all three datasets.Edit-distance and LR models exploit surface word features.
- 4.2.2 Model Performance and Dataset Difference: MultiP performs best on PIT-2015 but drops on MSRP and URL because its shared-topic constraint is approximated with named entities there.Its assumption matches PIT-2015's naturally shared topical phrases more closely.
- 4.2.2 Model Performance and Dataset Difference: LEX-WMF and LEX-OrMF show consistently high performance across datasets, close to the more complicated DeepPairwiseWord model.DeepPairwiseWord is more helpful on the two Twitter datasets, which contain lexically divergent paraphrases.
5 Extracting Phrasal Paraphrases
The authors extract phrasal paraphrases from Twitter URL-linked sentence pairs using word alignment and rank them with language, translation, embedding, and supervised scores. The resulting resource complements PPDB while yielding up-to-date paraphrase candidates.
- 5 Extracting Phrasal Paraphrases: 114,025 sentential paraphrases were harvested from raw 1% Twitter data with average precision = 69.08% using LEX-OrMF.The model accepted system predictions with probability ≥0.5, enabling continuous harvesting and subsequent lexical and phrasal paraphrase extraction.
- 5.1 Phrase Extraction and Ranking: 245,686 phrasal paraphrases were extracted with the GIZA++ word aligner from the Twitter URL corpus.The authors also explored the Jacana and Md Sultan supervised monolingual aligners.
- 5.1 Phrase Extraction and Ranking: Phrase pairs were ranked using language-model, translation, GloVe, and supervised scores.The supervised SVM regression model used human-rated phrase pairs and combined these scores with additional Moses features.
- 5.2 Paraphrase Quality Evaluation: The ranking methods were compared with human judgments on 200 phrase pairs sampled from GIZA++.Figure 3 evaluates automatic scores against 5-point human ratings.
- 5.2 Paraphrase Quality Evaluation: Twitter URL data has little overlap with PPDB, with only 1.3% and 0.8% overlap components reported.The authors conclude that the resource complements PPDB, which is primarily derived from well-edited texts.
6 Related Work
Earlier sentential paraphrase resources came from aligned news, translated texts, definitions, video descriptions, or rephrased sentences, but these sources do not scale readily. Existing phrasal resources and Twitter studies address related problems through different data types and collection strategies.
- Sentential Paraphrase Data: Earlier sentential paraphrase datasets used multiple reports, translations, definitions, video descriptions, or rephrased sentences.These collection methods were incapable of obtaining sentential paraphrases on a large scale.
- Non-sentential Paraphrase Data: Existing phrasal and syntactic resources include DIRT, POLY, PATTY, DE-FIE, and PPDB, mostly built from news or web data.Earlier Twitter paraphrase extraction used unsupervised approaches or small datasets.
7 Conclusion and Future Work
The paper concludes that a simple Twitter-based method can continuously collect large-scale sentential paraphrases. It releases a manually labeled dataset and automatically labeled monthly growth while identifying multilingual expansion and language-independent identification as future directions.
- The collection method is simple, effective, and continuously extracts large-scale sentential paraphrases from Twitter.
- 51,524 manually labeled sentence pairs form the paper’s new dataset, which the authors will share with the research community.
- 30,000 sentential paraphrases are added monthly through automatic labeling, enabling continuous corpus growth.
- Future work includes expanding to social-media languages and developing language-independent automatic paraphrase-identification models.