Source-linked AI summary
Learning Distributed Representations of Sentences from Unlabelled Data
Felix Hill, Kyunghyun Cho, Anna Korhonen
TL;DR
Learning sentence representations from unlabelled data lacks the consensus available for word representations. The paper systematically compares methods and introduces FastSent and SDAE, finding that optimal models depend critically on whether representations support supervised prediction or direct distance-based querying.
Problem
There is limited consensus about which architectures and objectives best learn useful distributed phrase or sentence representations from unlabelled data.
Method
The paper systematically compares unsupervised sentence-representation methods and introduces FastSent and Sequential Denoising Autoencoders.
Results
Deeper models generally perform best for supervised systems, while shallow log-linear models work best when representation spaces are queried with cosine distance.
Takeaways & Limitations
The appropriate representation-learning objective depends critically on the intended application, with fast shallow BOW models sometimes achieving the best unsupervised performance.
Takeaways & Limitations
The study’s evaluations may share limitations: inter-rater agreement is reported for only one of eight evaluations, and some items receive only one or two ratings.
Abstract
from arXiv · showhide
Unsupervised methods for learning distributed representations of words are ubiquitous in today's NLP research, but far less is known about the best ways to learn distributed phrase or sentence representations from unlabelled data. This paper is a systematic comparison of models that learn such representations. We find that the optimal approach depends critically on the intended application. Deeper, more complex models are preferable for representations to be used in supervised systems, but shallow log-linear models work best for building representation spaces that can be decoded with simple spatial distance metrics. We also propose two new unsupervised representation-learning objectives designed to optimise the trade-off between training time, domain portability and performance.
1 Introduction
The paper addresses limited consensus about learning useful sentence representations from unlabelled data through a systematic comparison of methods and two new objectives. It finds that the best approach depends on how representations will be evaluated and used.
- Sentence representations lack the methodological consensus established for word representations, despite their importance in language-processing systems.
- The study compares cutting-edge sentence-representation methods that do not require task-specific labelled data.The authors motivate this restriction as more cost-effective and applicable across languages and domains.
- The authors propose Sequential Denoising Autoencoders and FastSent as new unsupervised phrase- or sentence-representation objectives.
- The evaluation distinguishes supervised tasks, where representations feed labelled classifiers or regressors, from unsupervised tasks, where cosine distance directly queries representation spaces.
- Deeper models generally perform best in supervised evaluations, whereas shallow log-linear models generally perform best on unsupervised benchmarks.SkipThought leads most supervised evaluations, SDAE leads paraphrase identification, and FastSent leads the unsupervised SICK benchmark.
2 Distributed Sentence Representations
The study compares text-based, bottom-up, structured-resource, multimodal, and translation-based approaches, while introducing FastSent and SDAE to reduce training or data-order constraints. These models differ in their inputs, architectures, resource requirements, and sentence-encoding objectives.
- Scope: The comparison focuses on neural language models that learn sentence representations from naturally occurring unlabelled data.This scope excludes innovative supervised sentence-level architectures and most bottom-up composition methods.
- Existing text-based models: SkipThought encodes a sentence with an order-sensitive RNN and trains decoders to predict the preceding and following sentences.Its training cost sums the negative log-likelihoods of the correct words in both target sentences.
- Existing text-based models: ParagraphVector learns sentence vectors by predicting words from sentence representations, optionally combining them with consecutive-word embeddings.
- Comparison baselines: The study also evaluates additive CBOW or SkipGram composition, TFIDF BOW, dictionary-definition models, caption-based multimodal models, and neural machine-translation representations.These approaches use raw text, fixed word-vector operations, structured resources, captions, or aligned translated text.
- Novel text-based models: SDAE corrupts sentences by deleting words or swapping bigrams, then trains an LSTM encoder-decoder to reconstruct the original sentence.Unlike SkipThought, SDAE can train on sentences in arbitrary order, avoiding a coherent inter-sentence narrative requirement.
- Novel text-based models: FastSent uses an additive bag-of-words representation to predict adjacent sentences, targeting SkipThought’s signal at lower computational expense.Its source sentence vector is formed by summing source word embeddings; FastSent+AE additionally predicts the source sentence’s own words.
3 Evaluating Sentence Representations
The paper evaluates sentence representations in both supervised classification and unsupervised relatedness settings, reflecting whether vectors support learned downstream tasks or direct spatial comparison.
- Evaluation design: The study compares representation methods using evaluation setups that distinguish learned task performance from direct retrieval or comparison.These settings correspond to injecting general knowledge into supervised models versus querying the representation space directly.
- Supervised evaluation: Supervised evaluation measures representation utility by training classifiers on six sentence classification tasks.The tasks include paraphrase identification, sentiment, subjectivity, opinion polarity, and question type classification.
- Unsupervised evaluation: Unsupervised evaluation compares cosine distances between sentence vectors with human semantic-relatedness judgments.The evaluation uses SICK and STS 2014 sentence-pair datasets.
- Unsupervised evaluation: The SICK dataset contains 10,000 sentence pairs, while STS 2014 contains 3,750 pairs spanning six linguistic domains.SICK trial pairs are held out for tuning hyperparameters before applying the selected settings to both evaluation types.
4 Results
Results differ by evaluation type: several models excel on supervised tasks, while simple distributional approaches perform strongly on unsupervised relatedness benchmarks. Across both settings, BOW DictRep with pre-trained word embeddings is the most consistent performer.
- Supervised evaluations: SkipThought vectors perform best on three of six supervised evaluations, BOW DictRep with pre-trained embeddings on two, and SDAE on one.SDAE exceeds SkipThought by three percentage points on paraphrase identification.
- Supervised evaluations: SDAE performs notably well on paraphrase identification and is consistently better than SAE.Its paraphrase performance approaches state-of-the-art models designed specifically for that task.
- Unsupervised evaluations: BOW DictRep performs best on four of six STS categories and overall, and is joint-top on SICK.Among raw-text models, adding CBOW word vectors works best on STS.
- Unsupervised evaluations: FastSent is the best raw-text model on SICK, nearly matching C-PHRASE while using less than a third of the training text and no supervised syntactic representations.This result supports the value of sentence-level distributional information from adjacent context.
- Overall comparison: BOW DictRep with pre-trained word embeddings shows the most consistent performance across supervised and unsupervised evaluations.The authors suggest that its robustness may make it useful when the eventual application is nonspecific or unknown.
5 Discussion
The discussion shows that representation quality depends on the evaluation and intended application: different objectives produce different spaces, and supervised strengths do not reliably transfer to unsupervised distance-based evaluation. It also highlights benchmark consistency and practical resource constraints as important qualifications.
- Different learning methods are preferable for different intended applications, with greater variation than for word representations.
- CBOW, SDAE, DictRep, and ParagraphVec produce neighbours with substantial word overlap, whereas SkipThought and FastSent often match general concepts or sentence function with minimal overlap.
- FastSent’s representation-space behaviour may matter more than its marginal SICK improvement, while CBOW and FastSent also support fast online nearest-neighbour querying.
- SkipThought, SDAE, and NMT representations can perform well under supervised evaluation yet poorly when their geometry is decoded with cosine distance.Their training costs use nonlinear decoding, so informative geometry may not be reflected in simple cosine distance; log-linear models generally perform better unsupervised.
- 76.3 versus 76.6 are the average supervised scores for word-order-sensitive versus order-insensitive models, while unsupervised averages favour BOW models, 0.55 versus 0.42 for RNNs.The discussion suggests either that order-critical sentences are relatively rare or that current evaluations inadequately reflect order-dependent meaning.
- The evaluations’ overall internal consistency is 0.81, but the benchmarks may share limitations, including sparse inter-rater reporting and subjective sentential relatedness.Consistency is higher when supervised and unsupervised tasks are treated as separate cohorts, indicating that they prioritise different representation properties.
6 Conclusion
The paper finds substantial performance variation across evaluation settings and concludes that the best sentence-representation approach depends critically on whether use is supervised or unsupervised. It also identifies task-specific strengths for FastSent and Sequential Denoising Autoencoders, while noting a possible all-round choice when the application is unknown.
- The optimal sentence-representation approach depends critically on whether representations will be used in supervised or unsupervised settings.
- FastSent and Sequential Denoising Autoencoders perform particularly well on specific tasks, including MSRP and SICK sentence relatedness, respectively.
- When the application is unknown, DictRep may be the best all-round choice.
- The authors provide training and evaluation code, pre-trained models, and an online demo of the FastSent sentence space.
- Future work will consider supervised architectures and training across multiple supervised tasks to induce more general knowledge.