Source-linked AI summary
Siamese CBOW: Optimizing Word Embeddings for Sentence Representations
Tom Kenter, Alexey Borisov, Maarten de Rijke
TL;DR
Existing word embeddings are not optimized for sentence representation, despite averaging being an efficient way to obtain sentence embeddings. Siamese CBOW directly trains embeddings for averaging by predicting neighboring sentences, and it outperforms word2vec and skip-thought baselines on 14 of 20 test sets. The paper concludes that the method robustly generates high-quality sentence representations, while comparisons with Hill et al. require caution because vocabulary coverage may differ.
Problem
Existing word embeddings are not optimized specifically for representing sentences, although averaging them is an effective sentence-embedding method.
Method
Siamese CBOW directly optimizes word embeddings for averaging by predicting adjacent sentences from averaged sentence representations using cosine similarity and softmax.
Results
14 out of 20 test sets show Siamese CBOW outperforming word2vec and skip-thought baselines.
Takeaways & Limitations
Siamese CBOW provides a robust way to generate high-quality sentence representations across varied settings.
Takeaways & Limitations
Comparisons with Hill et al. should be interpreted cautiously because the experiments may use different vocabulary coverage.
Abstract
from arXiv · showhide
We present the Siamese Continuous Bag of Words (Siamese CBOW) model, a neural network for efficient estimation of high-quality sentence embeddings. Averaging the embeddings of words in a sentence has proven to be a surprisingly successful and efficient way of obtaining sentence embeddings. However, word embeddings trained with the methods currently available are not optimized for the task of sentence representation, and, thus, likely to be suboptimal. Siamese CBOW handles this problem by training word embeddings directly for the purpose of being averaged. The underlying neural network learns word embeddings by predicting, from a sentence representation, its surrounding sentences. We show the robustness of the Siamese CBOW model by evaluating it on 20 datasets stemming from a wide variety of sources.
1 Introduction
Siamese CBOW addresses the lack of sentence-specific optimization in word embeddings by training embeddings to be averaged into sentence representations and evaluating them broadly.
- Averaging word embeddings is an effective sentence-representation baseline, but existing embeddings are not optimized specifically for sentence representation.
- Siamese CBOW directly optimizes word embeddings for producing sentence representations while retaining general-purpose training from unlabeled data.
- The model adapts word2vec’s surrounding-word prediction strategy to predict sentences adjacent to one another in documents.
- The paper tests whether this direct optimization improves averaged sentence embeddings over word2vec across 20 evaluation sets from varied sources.
- Siamese CBOW is presented as an efficient architecture whose embeddings are evaluated across varied datasets to assess robustness across settings.
2 Siamese CBOW
Siamese CBOW learns sentence-oriented word embeddings by averaging shared word vectors and predicting neighboring sentences with a similarity-based objective.
- Training objective: The training criterion assigns higher probability to sentences observed adjacent to an input sentence and uses randomly selected nonadjacent sentences as negatives.
- Training objective: The network minimizes categorical cross-entropy between target adjacency probabilities and predictions computed from sentence embeddings.
- Training objective: With two positive and two negative examples, the target distribution is (0.5, 0.5, 0, 0).
- Siamese CBOW computes sentence embeddings by averaging constituent word embeddings, then compares sentences using cosine similarity and softmax probabilities.
- Network architecture: The word embedding matrix is shared across inputs and provides the network’s only trainable parameters, updated with stochastic gradient descent.
3 Experimental Setup
The experiments directly evaluate Siamese CBOW and two baselines on 20 SemEval datasets using cosine similarity between sentence representations. The setup controls training data and preprocessing while reporting Pearson’s r and Spearman’s r.
- Systems: Siamese CBOW, word2vec, and skip-thought are compared by computing cosine similarity between the sentence embeddings they produce.Siamese CBOW and word2vec average word embeddings, whereas skip-thought uses recurrent neural networks that account for word order.
- Training conditions: All models are trained or evaluated on the same Toronto Book Corpus data without extra features, preprocessing, or supervised training.The corpus contains 74,004,228 sentences and 1,057,070,918 tokens; the experimental vocabulary retains words appearing at least five times.
- Baseline considerations: The word2vec baseline uses both Skipgram and CBOW architectures with 300-dimensional embeddings and default settings.Its settings include minimum word frequency 5, context window 5, and five negative examples.
- Baseline considerations: Skip-thought has a vocabulary of 930,913 words, three times larger than the vocabulary used for Siamese CBOW and word2vec.The authors note this gives skip-thought an advantage when encoding sentences.
- Datasets and metrics: 20 SemEval datasets provide sentence pairs from varied sources, including newswire, tweets, and video descriptions, with human similarity annotations.The annotations use a 5-point scale, and the evaluation uses Pearson’s r and Spearman’s r.
- Training conditions: Siamese CBOW is trained for one epoch with 300-dimensional embeddings, two negative examples, batch size 100, and initial learning rate 0.0001.The network is implemented in Theano and trained on GPUs in the DAS5 cluster.
4 Results
Siamese CBOW outperforms baseline systems on most evaluation datasets, while remaining stable across training and parameter settings. Its averaging-based sentence representations also offer linear-time, prediction-efficient computation.
- 4.1 Main experiments: 14 out of 20 SemEval datasets show Siamese CBOW outperforming the baseline systems.On 2014 tweet-news, it reaches 0.7315 Pearson’s r, close to the best SemEval run’s 0.792 despite being completely unsupervised.
- 4.1 Main experiments: Siamese CBOW is close to skip-thought on the longest-sentence dataset, while achieving the best performance on some low-overlap sets.It is close to skip-thought on 2013 SMT, and performs best on 2015 belief and 2012 MSRpar despite their limited lexical overlap.
- 4.1 Main experiments: FastSent and Siamese CBOW each outperform the other on half of the evaluation sets, suggesting complementary differences between the methods.The comparison should be interpreted cautiously because vocabulary coverage may differ between the experiments.
- 4.2.1 Performance across iterations: Performance varies little across five training epochs on most datasets, although 2014 deft-news, 2013 OnWN, and especially 2012 MSRvid are exceptions.The first two appear to stabilize by epoch 5; MSRvid’s decline may reflect its very short sentences and the oversized 300-dimensional representation.
- 4.2.2 Number of negative examples: One or two negative examples typically suffice, while higher counts offer limited gains at substantial computational cost.Ten negative examples occasionally help, such as on 2013 FNWN, but 1 or 2 can be markedly better, as on 2015 belief.
- 4.2.3 Number of dimensions: Moderate embedding dimensionality of 200 or 300 is preferred because 50 or 100 dimensions are inferior and larger vectors cost more computationally.Increasing dimensionality improves performance on some datasets, including 2014 deft-forum, 2015 answ-forums, and 2015 belief.
- 4.3 Time complexity: All evaluated algorithms have O(n) complexity in input terms, but averaging methods use fewer costly operations than recurrent skip-thought.Word2vec and Siamese CBOW require |T|−1 vector additions and one scalar multiplication, whereas skip-thought performs 6|T| matrix multiplications.
- 4.4 Qualitative analysis: Siamese CBOW is fast and efficient for computing sentence embeddings at prediction time.The authors therefore prefer simple averaging methods such as Siamese CBOW or word2vec when prediction speed is pivotal.
5 Related Work
Related approaches differ in their training targets, supervision, and computational costs. Siamese CBOW predicts surrounding sentences from averaged sentence representations rather than using multilingual reconstruction, explicit similarity pairs, or vocabulary-sized outputs.
- Comparison with multilingual models: Multilingual models predict the same sentence in another language, whereas Siamese CBOW predicts surrounding sentences in the same document.
- Comparison with similarity-based models: Wieting et al. require explicitly labeled semantic-similarity phrase pairs, while Siamese CBOW uses virtually unlimited unlabeled training data.The related method also uses a margin-based loss and computationally expensive batch-wide negative-example comparisons.
- Siamese CBOW: Siamese CBOW predicts a sentence from neighboring sentences using averaged word vectors, closely paralleling CBOW while directly comparing sentence representations.Unlike word2vec CBOW, it compares sentence representations rather than a partial sentence representation with a word representation.
- Other siamese architectures: Yih et al. represent short texts with tf-idf vectors and learn a linear combination through a two-layer fully connected network, unlike Siamese CBOW's averaged word embeddings.Their objective combines positive-negative similarity differences in a logistic loss.
- Comparison with Hill et al.: Hill et al.'s model has twice as many parameters and uses a vocabulary-sized softmax, whereas Siamese CBOW ties parameters and uses a softmax over positive and negative examples.Siamese CBOW's softmax size is unaffected by vocabulary growth, unlike Hill et al.'s vocabulary-sized output.
6 Conclusion
Siamese CBOW learns word embeddings from unlabeled text by predicting preceding and following sentences from an input sentence representation. Across 20 test sets, it outperforms the word2vec and skip-thought baselines in most cases, while supervised-task and larger-text applications remain open.
- 6 Conclusion: Siamese CBOW efficiently learns word embeddings optimized for sentence representations using only unlabeled text data.The model predicts the preceding and following sentence from an input sentence representation.
- 6 Conclusion: 14 out of 20 test sets show Siamese CBOW outperforming word2vec and skip-thought baselines.
- 6 Conclusion: Parameter analyses indicate stability across settings, supporting the conclusion that Siamese CBOW robustly generates high-quality sentence representations.
- Limitations and future work: The paper does not analyze how Siamese CBOW embeddings affect supervised tasks.The authors identify this as an open question because supervised methods were beyond the paper's scope.
- Limitations and future work: The evaluation uses sentence pairs, leaving performance on larger texts such as documents for future investigation.The authors state that no theoretical limitation restricts the method to sentences.