PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Universal Paraphrastic Sentence Embeddings

John Wieting, Mohit Bansal, Kevin Gimpel, Karen Livescu

arXiv:1511.08198v3cs.CLcs.LG

TL;DR

The paper asks how to learn general-purpose paraphrastic sentence embeddings that transfer across domains and tasks. It compares compositional architectures using PPDB supervision and finds a cross-task trade-off: word averaging is strongest for similarity, entailment, and broad transfer, whereas LSTMs are strongest for sentiment. The resulting embeddings are simple, efficient resources that can also support supervised models.

  • Problem

    The paper addresses the problem of learning general-purpose, paraphrastic sentence embeddings that work across domains and tasks.

  • Method

    The authors train PPDB-supervised sentence embeddings and compare six compositional architectures, then combine the embeddings with supervised NLP tasks as priors or fixed feature extractors.

  • Results

    Word averaging performs strongly across domains and on similarity and entailment, while the LSTM achieves 89.2% on coarse-grained sentiment classification and new state-of-the-art performance there.

  • Takeaways & Limitations

    PARAGRAM-PHRASE XXL provides a simple, efficient baseline for universal sentence embeddings and can improve supervised similarity and entailment models.

  • Takeaways & Limitations

    Many PARAGRAM-PHRASE embeddings are undertrained because only 7,113 of 37,366 unique PPDB XL tokens appear more than 100 times.

Abstract

from arXiv · show

We consider the problem of learning general-purpose, paraphrastic sentence embeddings based on supervision from the Paraphrase Database (Ganitkevitch et al., 2013). We compare six compositional architectures, evaluating them on annotated textual similarity datasets drawn both from the same distribution as the training data and from a wide range of other domains. We find that the most complex architectures, such as long short-term memory (LSTM) recurrent neural networks, perform best on the in-domain data. However, in out-of-domain scenarios, simple architectures such as word averaging vastly outperform LSTMs. Our simplest averaging model is even competitive with systems tuned for the particular tasks while also being extremely efficient and easy to use. In order to better understand how these architectures compare, we conduct further experiments on three supervised NLP tasks: sentence similarity, entailment, and sentiment classification. We again find that the word averaging models perform well for sentence similarity and entailment, outperforming LSTMs. However, on sentiment classification, we find that the LSTM performs very strongly-even recording new state-of-the-art performance on the Stanford Sentiment Treebank. We then demonstrate how to combine our pretrained sentence embeddings with these supervised tasks, using them both as a prior and as a black box feature extractor. This leads to performance rivaling the state of the art on the SICK similarity and entailment tasks. We release all of our resources to the research community with the hope that they can serve as the new baseline for further work on universal sentence embeddings.

1 INTRODUCTION

The paper develops general-purpose sentence embeddings by comparing compositional architectures trained with paraphrase supervision. It finds that simple word averaging transfers strongly across domains, while LSTMs excel particularly on sentiment classification.

  • Modeling approach: The study compares architectures ranging from word averaging to LSTMs for learning sentence embeddings from paraphrase supervision.The averaging model uses only learned word vectors, while added layers and recurrent variants increase compositional complexity.
  • Transfer evaluation: 16.5 Pearson’s r is the average margin by which word-averaging models beat the best LSTM model across 22 out-of-domain SemEval datasets.The datasets span news, tweets, web forums, and image and video captions.
  • Transfer evaluation: 17.1 points higher than averaging GloVe vectors and 12.8 points higher than averaging PARAGRAM-SL999 vectors is the average Pearson’s r achieved across all 22 SemEval datasets.The result favors embeddings learned specifically for vector averaging over simply averaging pretrained word embeddings.
  • Transfer evaluation: The final sentence embeddings ranked in the top 25% of submitted systems in every SemEval STS task from 2012 through 2015, being best or tied for best on 4 datasets.Those submitted systems were designed and tuned for the particular tasks.
  • Supervised evaluation: 89.2% on coarse-grained sentiment classification was achieved by the LSTM, which the authors report as new state-of-the-art performance on the Stanford Sentiment Treebank.Word-averaging models instead outperformed the LSTM on the supervised similarity and entailment tasks.
  • Using pretrained embeddings: PPDB-trained sentence embeddings improved supervised similarity and entailment models as a prior and performed favorably as fixed black-box features for downstream tasks.The prior-based approach rivaled state-of-the-art performance on the similarity and entailment tasks.
  • Resource release: PARAGRAM-PHRASE XXL is released as an efficient, easy-to-use resource because it consists merely of a new set of word embeddings.The authors propose it as a simple and strong baseline for universal sentence embeddings.

2 RELATED WORK

Prior work studies how to compose word vectors into representations for phrases, sentences, and longer sequences. This paper distinguishes its focus on domain-independent sequence embeddings from supervised compositional modeling tied to a training distribution.

  • Compositionality: Compositionality research asks how vectors for individual words should be combined into a vector representing a word sequence.Studies have compared operations such as vector addition for composing bigrams.
  • Compositionality: Proposed sequence-composition architectures include distributional-semantic methods, neural bag-of-words models, deep averaging networks, and feature-weighted averaging.The cited work also includes recursive neural networks and other neural architectures.
  • Supervised compositional modeling: Most prior compositional models are learned with supervised annotations and evaluated on test sets drawn from the same distribution as their training data.The composition function is optimized for objectives based on those annotations.
  • Domain-independent embeddings: This paper instead targets general-purpose, domain-independent embeddings for word sequences.Related approaches include autoencoders for sentences, paragraphs, or documents.
  • Learning formulation: The paper keeps its learning methodology mostly fixed while focusing its exploration on modeling choices.Its formulation draws on prior work that learns input representations by maximizing similarity in the learned space.

3 MODELS AND TRAINING

The paper compares six sentence-composition architectures trained on PPDB paraphrase pairs, ranging from word averaging to LSTMs. Training uses a margin-based objective with carefully selected or mixed negative examples.

  • Model architectures: Six models span increasing complexity from word averaging and projection models to DANs, RNNs, identity-RNNs, and LSTMs.The averaging model learns only word embeddings, while identity-RNNs begin as averaging models but can incorporate word order.
  • Model architectures: The word-averaging model represents a sequence by averaging its token vectors and learns no additional compositional parameters.The learned token embeddings are called PARAGRAM-PHRASE embeddings.
  • Model architectures: The identity-RNN is initialized to average word embeddings, then can model word order while regularization controls its deviation from averaging.Its matrices start as identity, its bias is zero, and its output is divided by sequence length.
  • Model architectures: LSTMs provide the most expressive architecture and are evaluated both with and without an output gate.The authors report that output-gate choice significantly affects performance, while all models train their word embeddings.
  • Training objective: All models train on PPDB phrase pairs with a margin loss that makes paraphrase pairs more similar than selected negative examples.Negative examples can be chosen by MAX similarity or by MIX, which combines MAX and random mini-batch sampling.

4.1 DATA

Evaluation covers 24 textual-similarity datasets spanning SemEval STS tasks and multiple domains, plus Twitter, semantic relatedness, and PPDB-based tasks. Tuning uses human-annotated PPDB phrase-pair datasets.

  • Evaluation datasets: The evaluation comprises 24 textual-similarity datasets covering SemEval STS tasks from 2012–2015 and several additional semantic-similarity tasks.Additional datasets include SemEval Twitter, Semantic Relatedness, and two PPDB-based tasks.
  • Evaluation datasets: STS asks systems to predict sentence similarity on a 0–5 scale, across datasets drawn from diverse textual domains.The scale ranges from different topics to completely equivalent sentences.
  • Evaluation datasets: The STS domains include news, image and video descriptions, lexical glosses, machine-translation evaluation, headlines, web forums, Twitter, and belief statements.These domains are distributed across the STS tasks from 2012 through 2015.
  • Tuning data: Hyperparameter tuning uses two human-annotated PPDB phrase-pair datasets: a 26,456-pair sample and a 1,000-pair challenging subset.The larger dataset is selected for tuning, although both datasets produce similar results in practice.

4.2 TRANSFER LEARNING

Transfer experiments compare the learned sentence embeddings with established baselines across SemEval textual-similarity datasets. Simple PARAGRAM-PHRASE and projection models perform strongly out of domain, while LSTMs are stronger on in-domain PPDB tasks.

  • Setup: The transfer setup trains on PPDB XL and evaluates on SemEval textual-similarity datasets without using STS data for tuning.Hyperparameters are selected using PPDB-based annotated data, and PARAGRAM-SL999 initializes the word embeddings.
  • Models and baselines: The comparison includes PARAGRAM-PHRASE, projection, DAN, identity-RNN, RNN, two LSTM variants, skip-thought vectors, averaged GloVe, and averaged PARAGRAM-SL999.Table 2 reports Pearson’s r × 100 alongside official task-ranking summaries.
  • Results: The simplest PARAGRAM-PHRASE and projection models outperform the other architectures on 17 of 22 datasets.Identity-RNN is next best, while LSTM models lag behind on these out-of-domain similarity datasets.
  • Results: In-domain PPDB tasks show similar performance across architectures, with LSTM models slightly outperforming the others.This contrasts with the out-of-domain results, where the simplest models are strongest.
  • Results: Removing the LSTM output gate improves textual-similarity performance on 18 of 22 datasets.The LSTM without an output gate also beats the PARAGRAM-SL999 baseline on 12 of 22 datasets.

4.3 PARAGRAM-PHRASE XXL

PARAGRAM-PHRASE XXL extends the strongest averaging model with more PPDB data and tuning, producing highly competitive textual-similarity performance while remaining efficient.

  • PARAGRAM-PHRASE XXL was trained for 10 epochs on 9,123,575 unique phrase pairs from the entire PPDB XXL phrase section.Hyperparameter tuning used all of PPDB XL before final training on PPDB XXL.
  • The PPDB task table reports Spearman’s ρ × 100 and includes both an oracle result and Annotated-PPDB-tuned test results for one task.The oracle is included because that dataset was used for model selection on the other tasks.
  • The model’s reported SemEval results use Pearson’s r×100, with bold scores marking results that match or exceed the best shared-task system.For 2014 SICK, summary statistics include only primary runs because the full result set was unavailable.
  • PARAGRAM-PHRASE XXL matches or exceeds the best performance on 4 datasets and comes within 3 points on 8 of 22 datasets.The matched datasets are SMT-news, SMT, deft forum, and belief.

4.4 USING REPRESENTATIONS IN LEARNED MODELS

The paper tests PPDB-trained sentence representations as initialization, regularization, and fixed features across similarity, entailment, and sentiment tasks. Word averaging is strongest for similarity and entailment, while LSTMs are strongest for sentiment, and representation transfer improves some supervised results.

  • 4.4 USING REPRESENTATIONS IN LEARNED MODELS: The experiments ask whether PPDB-trained embeddings improve supervised models through initialization and regularization or serve as downstream task features.The study evaluates these questions on SICK similarity, SICK entailment, and Stanford Sentiment Treebank classification.
  • 4.4 USING REPRESENTATIONS IN LEARNED MODELS: The models use task-specific output layers and losses: three-way softmax with negative log-likelihood for entailment and sigmoid-then-softmax binary classification for sentiment.Word embeddings receive L2 regularization toward their initial values rather than unrestricted L2 regularization.
  • 4.4 USING REPRESENTATIONS IN LEARNED MODELS: Word averaging outperforms the other architectures on similarity and entailment, whereas the LSTM is much stronger on sentiment classification.This pattern indicates that architecture rankings vary across evaluations.
  • 4.4 USING REPRESENTATIONS IN LEARNED MODELS: The supervised results are competitive with prior state-of-the-art benchmarks, including an LSTM sentiment result that surpasses the previous 88.1 score.The reported comparisons include 86.86 for SICK similarity, 85.1 for entailment, and 88.1 as the previous sentiment result.
  • 4.4.1 REGULARIZATION AND INITIALIZATION TO IMPROVE TEXTUAL SIMILARITY MODELS: Regularizing models toward universal PPDB parameters significantly improves similarity and entailment results but harms the LSTM’s sentiment performance.The improved similarity and entailment results are competitive with or better than the state of the art.
  • 4.4.2 REPRESENTATIONS AS FEATURES: With fixed PARAGRAM-PHRASE representations, projecting into 2400 dimensions improves similarity and entailment but does not benefit sentiment.The projected results outperform both types of skip-thought embeddings on the overlapping task.

5 DISCUSSION

The discussion tests why LSTMs underperform simpler models on similarity and entailment, examining sentence length, overfitting, tuning, data, and embedding coverage. It identifies under-trained and unknown-word embeddings as a clear improvement target.

  • LSTMs are outperformed by simpler models on the paper’s similarity and entailment tasks, motivating analysis of this unexpected trend.
  • Sentence length: At all sentence-pair lengths, PARAGRAM-PHRASE outperforms LSTM models, although LSTMs perform better on the shortest pairs.
  • Overfitting: 0.7535, 0.7572, 0.7565, and 0.7463 are the models’ respective PPDB training-data scores, indicating similar fit across architectures.
  • Tuning and optimization: The LSTM comparison was extensively tuned and matched or exceeded another published LSTM result on SICK, while models performed similarly on in-domain PPDB tasks.
  • Under-trained embeddings: 37,366 unique tokens occur in PPDB XL, but only 7,113 occur more than 100 times, leaving many PARAGRAM-PHRASE embeddings undertrained.
  • Under-trained embeddings: The fraction of words occurring fewer than 100 times in PPDB XL correlates negatively with performance deviation from the dataset 75th percentile, with Spearman’s ρ = -45.1.
  • Training data: Performance continually increases with more PPDB training data, and training on just 92 phrase pairs improves average performance by 4 points over PARAGRAM-SL999.

6 QUALITATIVE ANALYSIS

Qualitative analyses show that PARAGRAM-PHRASE reshapes nearest-neighbor structure and emphasizes content words, explaining much of its advantage while exposing a bag-of-words weakness.

  • Training-data analysis: Figure 1 plots average Pearson’s r across 22 textual similarity datasets against PPDB training-data size on a log scale, comparing Ordered and Random selection with embedding baselines.
  • Nearest neighbors: PARAGRAM-PHRASE neighbors generally show stronger paraphrasing relationships and fewer antonym or co-hyponym errors than PARAGRAM-SL999 neighbors.
  • Nearest neighbors: Agree remains the nearest neighbor of disagree because bag-of-words training makes oppositional phrases highly similar; contextual modeling could address this issue.
  • Embedding geometry: Content words such as poverty and humanitarian have the largest L2 norms, whereas function words such as of and the have the smallest.
  • Embedding geometry: 66.83 is the PARAGRAM-PHRASE average Pearson’s r, versus 54.94 for PARAGRAM-SL999 and 62.64 after scaling PARAGRAM-SL999 vectors by learned importance weights.
  • Embedding geometry: At least 64.76% of the improvement over PARAGRAM-SL999 is attributed to weighting tokens by their importance, while reciprocal frequency weighting reaches average Pearson’s r of 45.52.

7 CONCLUSION

The paper concludes that simple averaged PARAGRAM-PHRASE embeddings provide efficient, strong sentence representations across tasks and domains. They can also serve as priors or fixed features, while future work targets undertrained words and broader generalization.

  • PARAGRAM-PHRASE is proposed as a simple, efficient baseline that performs strongly across a broad range of tasks and domains without requiring a neural network architecture.
  • Sentence embeddings can be averaged directly for applications, used as priors for similarity and entailment models, or used as fixed representations in classifiers.
  • Future work focuses on handling undertrained words and developing models that generalize better across the large suite of text similarity tasks.
Loading 1511.08198v3…