Source-linked AI summary
Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features
Matteo Pagliardini, Prakhar Gupta, Martin Jaggi
TL;DR
Sentence-level semantic embeddings remain difficult to learn in a general-purpose unsupervised way. The paper introduces Sent2Vec, which combines word and n-gram embeddings in an efficient unsupervised objective, and reports stronger performance than competing unsupervised models across benchmark evaluations.
Problem
Learning useful general-purpose semantic embeddings for sentences and longer text remains challenging despite successful unsupervised word embeddings.
Method
Sent2Vec learns sentence embeddings by averaging word and n-gram embeddings and training them jointly with a negative-sampling objective.
Results
3.8 % points improvement over the state of the art is reported for models trained on the Toronto books dataset in the macro average comparison.
Takeaways & Limitations
The resulting embeddings are reported as generalizable, simple, interpretable, and efficient for training and inference.
Takeaways & Limitations
Future work includes augmenting the model to exploit data with ordered sentences and investigating pretrained embeddings for downstream transfer learning.
Abstract
from arXiv · showhide
The recent tremendous success of unsupervised word embeddings in a multitude of applications raises the obvious question if similar methods could be derived to improve embeddings (i.e. semantic representations) of word sequences as well. We present a simple but efficient unsupervised objective to train distributed representations of sentences. Our method outperforms the state-of-the-art unsupervised models on most benchmark tasks, highlighting the robustness of the produced general-purpose sentence embeddings.
1 Introduction
Sentence embeddings remain difficult to learn unsupervised despite advances in unsupervised word embeddings. The paper proposes Sent2Vec as a simple, scalable extension of C-BOW using word and n-gram features.
- Sentence-level semantic embeddings remain challenging to produce and learn, especially in a general-purpose unsupervised way.
- Averaged word vectors had already outperformed LSTMs for sentence embeddings, motivating a trade-off favoring simpler scalable models.
- Sent2Vec composes sentence embeddings from word vectors and n-gram embeddings while training the composition and embeddings jointly.
- O(1) vector operations per processed word support training on extremely large datasets in a streaming fashion and enable fast inference.
- The method reports significant improvements over current unsupervised and semi-supervised models, with robust general-purpose embeddings across prediction benchmarks.
2 Model
Sent2Vec extends C-BOW from fixed word contexts to entire variable-length sentences, averaging learned unigram and n-gram embeddings to predict words. Negative sampling, subsampling, dropout, and sparse updates support efficient scalable training.
- Sent2Vec extends C-BOW by using an entire sentence as context and optimizing word vectors for additive sentence composition.
- The sentence vector averages source embeddings for constituent words and n-grams listed in R(S).
- The unsupervised objective predicts target words from sentence representations using negative sampling to improve efficiency for the large vocabulary.
- Subsampling reduces frequent-word influence, while n-gram dropout and optional L1 regularization reduce overfitting and encourage sparsity.
- 2.2 Computational Efficiency: Inference and SGD training require |S| · h floating point operations per sentence, or |R(S)| · h with n-grams.
3 Related Work
Prior sentence-embedding methods range from ordered-sentence recurrent models and autoencoders to averaging, matrix-factorization, and syntactic approaches. Sent2Vec instead focuses on task-independent unsupervised learning with a simple sentence-level objective.
- Other approaches include ParagraphVector, static or weighted word averaging, denoising autoencoders, and syntactically informed C-PHRASE.
- Arora et al. represent sentences with weighted averages of pretrained word vectors and remove a common discourse component using principal-component post-processing.
- Huang and Anandkumar’s dictionary-learning step is task-specific, so it does not produce general-purpose embeddings.
- Several prior models use ordered corpora or neighboring sentences, including SkipThought, FastSent, and Siamese C-BOW.
- Sent2Vec is closest conceptually to FastText but predicts target words from source word sequences and averages source embeddings instead of summing them.
4 Evaluation Tasks
The evaluation measures sentence embeddings across supervised classification and unsupervised similarity benchmarks. Logistic regression supports supervised prediction, while cosine similarity is compared with human judgments using correlation metrics.
- The benchmark suite combines supervised and unsupervised tasks to assess generalization across domains and general-purpose sentence-embedding quality.
- Downstream Supervised Evaluation: Supervised evaluations use logistic regression on sentence embeddings for paraphrase, sentiment, product-review, subjectivity, and related classification tasks.
- Downstream Supervised Evaluation: For MSRP, features concatenate absolute representation differences with component-wise products, while other datasets directly use sentence embeddings.
- Unsupervised Similarity Evaluation: Unsupervised evaluation uses cosine similarity on STS 2014 and SICK 2014, compared with human judgments using Pearson’s r and Spearman’s ρ.
5 Results and Discussion
Sent2Vec performs strongly across supervised and unsupervised evaluations while remaining substantially faster and scalable than neural sentence-embedding approaches. Its results show broad generalization, with performance depending on dataset, task, and the inclusion of bigram features.
- Downstream Supervised Evaluation Results: Sent2Vec achieves state-of-the-art results on the CR supervised task and ranks second only to SkipThought on average across supervised evaluations.Its unigram-plus-bigram model is the best model after SkipThought on half of the supervised tasks, but it is weaker on MSRP.
- Unsupervised Similarity Evaluation Results: Sent2Vec is state-of-the-art on most Toronto-corpus unsupervised tasks and achieves the best averaged performance.It is on average at least as strong as C-PHRASE, despite lagging on the STS 2014 WordNet and News subtasks.
- Official STS 2017 benchmark: Sent2Vec significantly outperforms C-PHRASE on the official STS 2017 benchmark, delivering the best unsupervised baseline.This result is reported for the most recent edition of the benchmark discussed in the paper.
- Macro Average: 3.8 % points improvement over the state of the art is reported for models trained on the Toronto books dataset using the macro average.The macro average combines supervised and unsupervised task averages and includes training times.
- Efficiency and scalability: O(1) vector operations per processed word support streaming training on extremely large datasets and fast inference compared with neural approaches.The models are also reported to train faster than SkipThought and DictRep because SGD permits substantial parallelization.
- Model variants: Adding bigrams significantly improves supervised accuracy but generally lowers unsupervised similarity scores.The paper attributes this pattern to bigrams capturing non-compositional features while increasing representational uniqueness for similarity tasks.
6 Conclusion
The paper introduces Sent2Vec as an efficient unsupervised method for sentence embeddings and reports strong performance, generalizability, and interpretability. It identifies exploiting ordered sentences and pre-trained embeddings as future directions.
- Sent2Vec trains and infers sentence embeddings with a novel, computationally efficient, unsupervised C-BOW-inspired method.
- The model is reported as generalizable, extremely fast to train, simple to understand, and easily interpretable.
- Future work could augment Sent2Vec to exploit ordered sentences and investigate pre-trained embeddings for downstream transfer learning.
A Parameters for training models
The appendix identifies Table 5 as presenting the training parameters used for Sent2Vec models.
- Table 5 presents the training parameters for the Sent2Vec models.
B L1 regularization of models
The models optionally use L1 regularization with proximal-gradient soft thresholding to improve generalization and induce sparsity in word and n-gram vectors. The update adds only modest computational overhead while supporting larger embedding dimensions.
- Sparsity improves memory efficiency and permits higher embedding dimensions; τ is set to 0.0005 for both Wikipedia and Toronto Book Corpus unigram-plus-bigram models.
- L1 regularization is applied separately to source and target word and n-gram vectors, with τ denoting the regularization parameter.
- The proximal-gradient scheme performs a gradient step on f(z), then updates z using the proximal operator for the nondifferentiable regularizer g(z)=∥z∥1.
- Thresholding is applied after SGD to updated word and n-gram vectors, with separate parameters for source and target vectors based on τ, the learning rate, and sentence context size.
- L1 regularization gives a small performance boost, while thresholding costs |R(S \ {wt})| · h operations for word vectors and (|N| + 1) · h for target vectors.
C Performance comparison with Sent2Vec models trained on different corpora
The section compares Sent2Vec variants with semi-supervised and supervised models across downstream supervised tasks and unsupervised correlation evaluations. The tables identify dataset-best results and Sent2Vec performances matching or exceeding non-Sent2Vec models.
- Table 6 compares different Sent2Vec models with semi-supervised and supervised models across downstream supervised evaluation tasks.
- Table 7 compares Sent2Vec models with semi-supervised and supervised models using Spearman and Pearson correlation measures in unsupervised evaluation.
D Dataset Description
The comparison includes average sentence lengths for the datasets used in evaluation.
- Table 8 reports average sentence lengths for the datasets used in the comparison.