PaperlayerView exact revision on arXiv

Source-linked AI summary

Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, Antoine Bordes

arXiv:1705.02364v5cs.CL

TL;DR

Sentence-level representations had not matched the usefulness and adoption of word embeddings, particularly under unsupervised learning. The paper trains and evaluates supervised sentence encoders, finding that NLI-trained representations, especially a BiLSTM with max pooling, transfer better than unsupervised alternatives across 12 tasks. It concludes that NLI is suitable for learning generic sentence embeddings, while larger natural-language-understanding datasets may improve quality further.

  • Problem

    Sentence representations that capture full-sentence meaning had not achieved the clarity or performance of established word embeddings, especially with unsupervised methods.

  • Method

    The paper compares sentence-encoding architectures and trains universal sentence encoders using supervised natural language inference data, evaluating their transferability.

  • Results

    NLI-trained models perform better than models trained unsupervised or on other supervised tasks across 12 transfer tasks, with BiLSTM max pooling outperforming existing approaches such as SkipThought.

  • Takeaways & Limitations

    Natural language inference is a suitable supervised task for learning generic sentence representations transferable to other NLP tasks.

  • Takeaways & Limitations

    The paper notes that larger datasets relying on natural-language understanding for sentences could improve sentence-embedding quality.

Abstract

from arXiv · show

Many modern NLP systems rely on word embeddings, previously trained in an unsupervised manner on large corpora, as base features. Efforts to obtain embeddings for larger chunks of text, such as sentences, have however not been so successful. Several attempts at learning unsupervised representations of sentences have not reached satisfactory enough performance to be widely adopted. In this paper, we show how universal sentence representations trained using the supervised data of the Stanford Natural Language Inference datasets can consistently outperform unsupervised methods like SkipThought vectors on a wide range of transfer tasks. Much like how computer vision uses ImageNet to obtain features, which can then be transferred to other tasks, our work tends to indicate the suitability of natural language inference for transfer learning to other NLP tasks. Our encoder is publicly available.

1 Introduction

The paper addresses how to learn sentence representations that capture full-sentence meaning and transfer effectively across tasks. It compares supervised training and encoder architectures, finding strong transfer from an NLI-trained BiLSTM with max pooling.

  • Sentence representations must capture relationships among multiple words and phrases in a single vector, a problem less settled than learning useful word embeddings.
  • The paper studies sentence encoders trained on large corpora and transferred to other tasks, comparing architecture choices and supervised versus unsupervised training.
  • An encoder based on a bi-directional LSTM with max pooling, trained on SNLI, yields state-of-the-art sentence embeddings against unsupervised alternatives such as SkipThought and FastSent.
  • The finding is established across a broad and diverse set of transfer tasks measuring whether sentence representations capture general and useful information.

2 Related work

Prior sentence-representation work largely relied on unsupervised objectives because the best supervised training task remained unclear and specialization could discard general sentence information. Earlier supervised comparisons underperformed, whereas fully exploiting SNLI produced consistent gains over unsupervised approaches.

  • Transfer learning has succeeded in computer vision, where ImageNet-trained image features support tasks including face recognition and visual question answering.
  • Most sentence-representation approaches are unsupervised because NLP had not identified a clearly suitable supervised task for whole-sentence semantics.
  • Unsupervised sentence methods include compositional models and SkipThought, which predicts neighboring sentences and demonstrated good performance on 8 transfer tasks.
  • Earlier supervised models trained on machine translation, dictionary-definition, and image-captioning data obtained significantly lower results than SkipThought.
  • This work fully exploits SNLI for generic sentence encoders and consistently outperforms unsupervised approaches despite using much less human-annotated data.

3 Approach

The approach trains generic sentence encoders on natural language inference and compares recurrent, self-attentive, and convolutional architectures. It uses separate sentence encodings for NLI and evaluates multiple pooling and composition strategies.

  • 3.1 The Natural Language Inference task: SNLI provides 570k human-generated English sentence pairs labeled entailment, contradiction, or neutral for supervised sentence-encoder training.The paper hypothesizes that NLI's semantic nature can produce universally useful sentence representations.
  • 3.1 The Natural Language Inference task: The selected NLI setup separately encodes premise u and hypothesis v, then combines them using concatenation, element-wise product, and absolute difference.The combined vector is used to predict the NLI relation.
  • 3.2 Sentence encoder architectures: Seven architectures are compared, including LSTM and GRU encoders, BiGRU-last, BiLSTMs with mean or max pooling, a self-attentive network, and hierarchical convolutional networks.The comparison targets which architecture best captures generically useful information.
  • 3.2.2 BiLSTM with mean/max pooling: BiLSTM encoders concatenate forward and backward hidden states, then form fixed-size sentence vectors through either dimension-wise max pooling or mean pooling.The bidirectional states read the sentence in opposite directions before pooling.
  • 3.2.3 Self-attentive network: The self-attentive encoder scores BiLSTM hidden states against learned context queries and combines them into weighted sentence representations.Multiple context vectors generate four representations that are concatenated.
  • 3.2.4 Hierarchical ConvNet: The hierarchical ConvNet uses four convolutional layers with max pooling and concatenates representations from different abstraction levels.This fixed-size representation is designed to capture hierarchical sentence abstractions.

4 Evaluation of sentence representations

The evaluation uses SentEval-style transfer experiments across classification, entailment, semantic relatedness, textual similarity, paraphrase detection, and caption-image retrieval. Performance is summarized across transfer tasks and compared across sentence encoder architectures.

  • Evaluation procedure: The evaluation measures sentence representations as features across 12 transfer tasks spanning classification, semantic matching, and multimodal retrieval.The tasks include binary and multiclass classification, SICK entailment and relatedness, STS14, paraphrase detection, and caption-image retrieval.
  • Binary and multi-class classification: Binary and multiclass classification covers sentiment, question type, product reviews, subjectivity, and opinion polarity using logistic regression on sentence vectors.The classifier uses a linear model with an L2 penalty tuned by grid search on validation data.
  • Entailment and semantic relatedness: SICK evaluation tests entailment and semantic relatedness with logistic regression over joint representations, reporting Pearson correlation for relatedness.Relatedness prediction models the probability distribution of scores rather than a single class.
  • STS14 - Semantic Textual Similarity: STS14 evaluates cosine-based sentence similarity across six unsupervised SemEval tasks covering news, forums, image descriptions, and headlines.Sentence pairs are labeled with similarity scores from 0 to 5.
  • Caption-Image retrieval: Caption-image retrieval ranks images for captions or captions for images using cosine similarity and reports Recall@K and median rank.The evaluation uses COCO with 113k training images, 5k validation images, and 5k test images.

5 Empirical results

Across transfer evaluations, supervised NLI training and architecture choice strongly affected sentence-representation quality. BiLSTM-max generally led, outperforming unsupervised and other supervised alternatives across diverse tasks, while larger or broader training data often improved transfer.

  • Architecture impact: BiLSTM-4096 with max pooling performed best on SNLI and transfer tasks, significantly exceeding the other evaluated architectures.The comparison includes LSTM, GRU, BiGRU-last, BiLSTM-Mean, inner-attention, and hierarchical-ConvNet models.
  • Architecture impact: Better performance on the training task did not necessarily translate into better transfer-task results.The paper specifically contrasts inner-attention and BiLSTM-Mean.
  • Optimization: Adam made BiLSTM-max converge in 5 epochs instead of 10 with SGD, but produced worse transfer results.The authors attribute this difference most likely to increased specialization on the training task.
  • Embedding size: Increasing embedding size improved performance for almost all models, particularly BiLSTM-Max, HConvNet, and inner-attention.The authors hypothesize these models incorporate information useful for transfer even when SNLI performance remains relatively stable.
  • Comparison with SkipThought: With 570k supervised SNLI sentences, BiLSTM-max consistently outperformed SkipThought vectors trained on 64M sentences across many transfer tasks.It also trained in less than a day on one GPU, compared with a month for the best SkipThought-LN network; on STS14, Pearson scores were 0.68 versus 0.29 and 0.44 for SkipThought variants.
  • NLI as a supervised training set: NLI-trained representations outperformed models trained on other supervised tasks and provided higher-quality features for both classification and similarity.The paper suggests NLI encourages encoding semantic information that is discriminative and informative.
  • Domain adaptation: On SICK, the transfer approach achieved Pearson 0.885 on SICK-R and 86.3% test accuracy on SICK-E, exceeding the cited previous results.The reported comparisons include 0.868 on SICK-R, 84.5% on SICK-E, and 80.8% for an earlier SNLI-pretrained transfer approach.
  • MultiGenre NLI: Training on SNLI and MultiNLI significantly boosted overall performance, reached AdaSent performance on CR, and achieved Pearson 0.70 on STS14.Performance on caption-related retrieval tasks did not degrade relative to SNLI-only training.

6 Conclusion

The paper evaluates supervised sentence embeddings on 12 transfer tasks and finds NLI-trained models, especially BiLSTM-max, outperform unsupervised and other supervised approaches. The authors position this result as evidence that NLI is useful for learning general sentence representations, while noting that larger datasets and other model-task combinations remain open opportunities.

  • 6 Conclusion: Across 12 transfer tasks, NLI-trained models outperformed models trained under unsupervised conditions or on other supervised tasks.The study also identified BiLSTM with max pooling as the strongest architecture among those explored.
  • 6 Conclusion: BiLSTM with max pooling produced the best current universal sentence-encoding methods and outperformed approaches such as SkipThought vectors.This conclusion follows the paper’s architecture comparisons and transfer evaluations.
  • 6 Conclusion: The authors state that the work explores only part of the possible combinations of models and tasks for generic sentence embeddings.They suggest larger datasets relying on natural-language understanding could further improve sentence-embedding quality.

Appendix

The appendix examines which BiLSTM hidden states max pooling selects before and after NLI training. Training changes pooling from a relatively even pattern to focused selection of words carrying much of a sentence’s meaning.

  • Max-pooling visualization: The appendix compares max-pooling visualizations for BiLSTM-max models trained on NLI and left untrained.The comparisons cover two pairs of entailed sentences.
  • Max-pooling visualization: For each time step, the analysis counts how often max pooling selects that hidden state as the sentence representation centered around its word.This provides the basis for comparing selection patterns across trained and untrained models.
  • Max-pooling visualization: Without training, max pooling is relatively even across hidden states but consistently favors the first and last states.The trained model instead focuses on specific words carrying most of the sentence’s meaning, without an explicit attention mechanism.
  • Max-pooling visualization: Although trained pooling focuses selectively, each hidden state incorporates sentence information at different levels, so the model still uses information from all hidden states.This explains why focused selection does not reduce the representation to isolated token information.
Loading 1705.02364v5…