Source-linked AI summary
An efficient framework for learning sentence representations
Lajanugen Logeswaran, Honglak Lee
TL;DR
Sentence-representation methods based on word-level context reconstruction can encode irrelevant surface details and train slowly. The paper introduces Quick Thoughts, which selects correct context sentences from candidates using sentence embeddings, and reports richer representations, state-of-the-art unsupervised results across several semantic tasks, and an order-of-magnitude reduction in training time.
Problem
Prior sentence-representation models reconstruct context sentence surface forms and incur high computational cost from sequential decoding and vocabulary-wide softmax layers.
Method
Quick Thoughts replaces word-level generation with classification that identifies the correct adjacent context sentence from candidate sentences using sentence embeddings.
Results
The representations outperform prior unsupervised and supervised methods on several downstream semantic tasks while consuming an order of magnitude less training time.
Takeaways & Limitations
The framework provides efficient learning of generic sentence representations from large unlabelled text corpora.
Abstract
from arXiv · showhide
In this work we propose a simple and efficient framework for learning sentence representations from unlabelled data. Drawing inspiration from the distributional hypothesis and recent work on learning sentence representations, we reformulate the problem of predicting the context in which a sentence appears as a classification problem. Given a sentence and its context, a classifier distinguishes context sentences from other contrastive sentences based on their vector representations. This allows us to efficiently learn different types of encoding functions, and we show that the model learns high-quality sentence representations. We demonstrate that our sentence representations outperform state-of-the-art unsupervised and supervised representation learning methods on several downstream NLP tasks that involve understanding sentence semantics while achieving an order of magnitude speedup in training time.
1 INTRODUCTION
The paper targets semantic sentence representations that avoid reconstructing irrelevant surface form while reducing the computational cost of context-prediction models. It proposes Quick Thoughts, a discriminative framework that learns from unlabelled text and reports faster training with stronger downstream performance.
- Existing encoder-decoder methods reconstruct context sentences, forcing representations to encode surface-form details irrelevant to sentence meaning.
- Sequential word decoding and vocabulary-wide softmax layers make prior sentence-representation methods computationally expensive.
- Quick Thoughts replaces generation with classification over candidate sentence embeddings, identifying the correct context sentence from contrastive alternatives.
- The framework is presented as a simple, general method that trains encoders an order of magnitude faster while achieving better performance.
- The paper reports new state-of-the-art unsupervised sentence representations across several downstream tasks involving sentence semantics.
2 RELATED WORK
Prior work learns sentence representations through supervised objectives, context prediction, reconstruction, coherence, or simpler bag-of-words methods. The paper positions its approach as combining flexible encoders with efficient training while relaxing the constraints of earlier coherence objectives.
- Skip-thought and related encoder-decoder models encode sentences and sequentially predict or reconstruct neighboring sentences.
- Autoencoder approaches reconstruct sentences and avoid requiring ordered sentences, while bag-of-words methods train efficiently by ignoring word order.
- Context-identification methods and FastSent use candidate selection or word prediction to learn sentence representations efficiently.
- Unlike binary coherent-versus-incoherent classification, the paper’s objective only requires observed contexts to be more plausible than contrastive alternatives.
- The proposed framework retains flexible encoder architectures while still enabling efficient training.
- Supervised natural language inference trains sentence encoders to distinguish entailment, neutrality, and contradiction between sentence pairs.
3 PROPOSED FRAMEWORK
The proposed framework predicts adjacent-sentence meaning in embedding space rather than reconstructing words. It encodes an input sentence, scores candidate context sentences, and trains the model to select the correct context.
- Quick Thoughts uses an embedding-space objective to predict the meanings of adjacent sentences instead of reconstructing their surface forms.
- Given an input sentence, the model chooses the correct target sentence from candidate sentences as a discriminative approximation to generation.
- Candidate sets contain one ground-truth context sentence and multiple non-context sentences for classification.
- The training objective maximizes the probability of identifying correct context sentences across the training data.
- The scoring function is an inner product, which limits classifier complexity and encourages the encoders to learn useful representations.
- At test time, the sentence representation concatenates the outputs of two separately parameterized encoders.
- The framework supports flexible encoders, including GRUs whose final hidden states represent sentences.
4 EXPERIMENTAL RESULTS
The experiments evaluate whether the learned representations capture sentence semantics in downstream NLP tasks. They use standard downstream evaluation and train models on large ordered-sentence corpora.
- The evaluation uses sentence representations as features for downstream NLP tasks focused on sentence semantics.
- The authors retain traditional downstream-task evaluation while acknowledging limitations of these evaluations.
- Models are trained on BookCorpus, containing about 45M ordered sentences, and the larger UMBC corpus, containing 129M sentences.
4.3 TRAINING
Training uses contiguous sentence sets to form contrastive candidate pools for context prediction, with validation accuracy guiding key hyperparameters.
- Each minibatch uses a contiguous set of corpus sentences as the candidate pool for classifying each sentence’s context.
- The contiguous-sentence scheme performed as well as random sampling and nearest-neighbor contrastive sampling.
- Validation prediction accuracy selected the batch size, learning rate, and prediction context size.
- The prediction context size was set to 3.
4.4 EVALUATION
The evaluation compares QT sentence representations with unsupervised, supervised, and task-specific systems across semantic benchmarks and image-caption retrieval. QT models generally match or exceed prior methods while reducing training cost, and larger data further improves performance.
- Comparison against unsupervised methods: QT variations perform comparably or better than skip-thought and CNN-based models across all tasks despite requiring much less training time.
- Comparison against unsupervised methods: The MultiChannel-QT model outperforms all previous methods, including a prior model using pre-trained word embeddings.
- Comparison against unsupervised methods: Pooling BookCorpus and UMBC data yields strict improvements on a majority of tasks over BookCorpus-only QT models.The pooled corpus contains approximately 174M sentences.
- Computational efficiency: MC-QT trains in just under 11 hours on BookCorpus, compared with two weeks reported for skip-thoughts.
- Computational efficiency: The RNN implementation fits within 3GB of GPU memory and supports larger vocabulary sizes than most previous models.
- Comparison against supervised methods: The ensemble approaches the best supervised task-specific methods and outperforms them on 3 of 8 tasks.
- Image-sentence ranking: On MSCOCO image-caption retrieval, QT significantly outperforms previous unsupervised pre-training methods in median retrieval rank for both annotation and search.
- Nearest neighbors: Nearest-neighbor examples often show QT retrieving sentences more related to queries than skip-thought, including reordered clauses with similar meaning.
5 CONCLUSION
The framework learns generic sentence representations efficiently from large unlabelled corpora, achieving richer representations with substantially less training time and strong downstream performance.
- The framework learns generic sentence representations efficiently from large unlabelled text corpora.
- The learned representations are richer than prior unsupervised and supervised methods while consuming an order of magnitude less training time.
- The approach establishes a new state-of-the-art for unsupervised sentence representation learning across several downstream tasks.
A ANALOGY MAKING
The analogy task evaluates whether sentence embeddings support relational retrieval. The model outperforms skip-thought vectors across several categories, but both methods share a common failure case.
- Analogy task formulation: Sentence analogies are formulated as retrieval using v(C) + v(B) − v(A) to identify the closest candidate sentence vector.
- Dataset construction: The dataset contains 1300 sentence analogy tuples, with four questions derived from each tuple and a candidate pool augmented by 1M other sentences.
- Quantitative results: The model outperforms skip-thought vectors across several word-pair categories and performs well on family and verb transformation categories.
- Qualitative results: Qualitative retrieval results show that both methods commonly fail when the model treats A and B as identical and retrieves sentence C.
- Scope: The evaluated transformations are mostly syntactic and involve only a few words, leaving higher-level multiword transformations for future exploration.
B SEMANTIC TEXTUAL SIMILARITY
Semantic textual similarity evaluation compares sentence-vector cosine similarities with human judgments. RNN and BoW encoders trained with the objective perform strongly against prior representation-learning methods.
- Evaluation: STS14 evaluates representations by correlating cosine similarity between sentence vectors with human similarity judgments.
- Models: The study evaluates RNN and BoW encoders trained with the proposed objective from scratch on BookCorpus.
- BoW training: Training the BoW model uses validation-selected embedding size, contrastive-sentence count, and context size, and takes 2 hours on a Titan X GPU.
- Results: The RNN-based encoder performs strongly against other sequence encoders, while the BoW variation performs comparably to prior BoW models.
- Results: Table 8 reports Pearson scores for QT RNN and BoW models alongside SDAE, CBOW, Skipgram, FastSent, Skip-Thoughts, and Siamese CBOW baselines.
C TRAINING EFFICIENCY
Training efficiency is evaluated by comparing identical encoder architectures trained with the proposed and skip-thought objectives over time. The proposed objective reaches downstream performance faster, with a reported 4.8x overall speedup.
- Experimental setup: The experiment trains the same single-layer GRU encoder architecture with the proposed and skip-thought objectives and compares downstream performance after different training durations.
- Parameterization: The proposed objective uses approximately 19.8M parameters, compared with approximately 57.7M for skip-thought.
- Evaluation setup: Evaluation uses only the input-side encoder parameters, approximately 9.9M parameters, for the compared representations.
- Results: The proposed objective shows apparent speed benefits in downstream-task comparisons using 1000-dimensional sentence embeddings.
- Results: 4.8x overall training speedup is observed for the proposed objective, although the output encoder is discarded in this experiment.
D REPRESENTATION SIZE, TRAINING EFFICIENCY AND PERFORMANCE
The experiments vary sentence-embedding size to examine the trade-off between training efficiency and representation quality. Smaller embeddings improve efficiency with only marginal performance loss in most cases, while selected reduced-size models outperform larger prior systems.
- Evaluation setup: The experiments use MC-QT models trained on BookCorpus and report mean accuracy across MSRP, TREC, MR, CR, SUBJ, and MPQA.Training times assume the two MC-QT component models are trained in parallel.
- Representation size and efficiency: Reducing embedding size improves training efficiency at the expense of marginal performance loss in most cases.The reported performance averages classification benchmarks, but differing classifier sizes make strong quality comparisons across embedding sizes difficult.
- Comparison with prior models: 1600-dimensional and 3200-dimensional models outperform the 4800-dimensional Skip-thought and Combine-CNN models on mean benchmark performance.Skip-thought and Combine-CNN achieve mean accuracies of 83.75 and 85.33, respectively.