Source-linked AI summary
Evaluation of sentence embeddings in downstream and linguistic probing tasks
Christian S. Perone, Roberto Silveira, Thomas S. Paula
TL;DR
Sentence embedding research lacked comprehensive evaluations across rapidly developing methods, limiting comparisons of their transfer and linguistic properties. The paper evaluates recent techniques across downstream and probing tasks, finding that ELMo-based bag-of-words performs strongly while no method transfers consistently across all tasks.
Problem
Comprehensive common-ground evaluation of rapidly developing sentence embedding methods remains limited, despite the goal of universal transfer across many downstream tasks.
Method
The paper evaluates multiple sentence embedding techniques using SentEval downstream tasks and 10 linguistic probing tasks.
Results
A bag-of-words approach using context-dependent ELMo word embeddings achieves excellent performance on many downstream tasks and captures important linguistic properties.
Takeaways & Limitations
Linguistic probing helps explore which relationships and properties different representation levels capture, supporting more informed choices for particular tasks.
Takeaways & Limitations
No evaluated method performs consistently across all tasks, and the paper concludes that the field remains far from a universal sentence encoder.
Abstract
from arXiv · showhide
Despite the fast developmental pace of new sentence embedding methods, it is still challenging to find comprehensive evaluations of these different techniques. In the past years, we saw significant improvements in the field of sentence embeddings and especially towards the development of universal sentence encoders that could provide inductive transfer to a wide variety of downstream tasks. In this work, we perform a comprehensive evaluation of recent methods using a wide variety of downstream and linguistic feature probing tasks. We show that a simple approach using bag-of-words with a recently introduced language model for deep context-dependent word embeddings proved to yield better results in many tasks when compared to sentence encoders trained on entailment datasets. We also show, however, that we are still far away from a universal encoder that can perform consistently across several downstream tasks.
1 Introduction
The paper addresses the lack of comprehensive, common-ground evaluation for rapidly developing sentence embedding methods. It evaluates transfer to downstream tasks and linguistic properties, showing that universal sentence representations remain elusive.
- Motivation: Sentence representations aim to transfer effectively across a wide variety of downstream tasks, but consistent broad performance remains unresolved.The paper describes this goal as the universal sentence encoder objective.
- Approach: Bag-of-words sentence representations can improve beyond traditional averaging through weighted averages, SVD modification, and concatenated power means.SIF is described as a strong baseline, while p-mean improves over SIF and traditional averaging.
- Approach: Encoder-based alternatives include self-supervised Skip-Thought and supervised InferSent trained on entailment data.InferSent uses a bi-directional LSTM and the Stanford Natural Language Inference dataset.
- Motivation: Recent sentence embedding techniques lack extensive evaluations covering them on common grounds.SentEval was proposed to reduce this comparison gap through a common evaluation pipeline.
- Approach: The study evaluates recent sentence embeddings for downstream transfer and linguistic properties using SentEval-based protocols.The evaluation creates a panorama of current techniques and compares performance across different tasks.
2 Related Work
Related work spans contextual and non-contextual word embeddings, sentence encoders, and aggregation methods. The paper positions its evaluation against unresolved questions about how sentence meaning should be represented.
- Word embeddings: Word embedding methods range from neural language models and Word2Vec to GloVe, FastText, and contextual ELMo representations.These approaches differ in how they model context, semantics, syntax, and subword information.
- Word embeddings: ELMo uses bi-directional LSTM language-model representations to capture context-dependent word meanings and syntactic information.Its token representations depend on the surrounding context and internal network layers.
- Open problem: Bag-of-words approaches showed good performance for some tasks, but there is no consensus on representing full sentence meaning.The paper identifies sentence representation as an open research problem.
- Sentence embeddings: Sentence methods include Skip-Thought encoder-decoder models, supervised InferSent, p-mean aggregation, and Universal Sentence Encoder variants.The Universal Sentence Encoder includes Transformer and deep averaging network encoders.
- Scope: The evaluation omits Doc2Vec/Paragraph2Vec, fastSent, and Sent2Vec because the selected methods were considered representative and suitable for indirect comparisons.This is an explicit scope choice in the paper's comparison set.
3 Evaluation tasks
The evaluation combines downstream-task transfer with linguistic probing to assess both task suitability and the linguistic properties captured by sentence embeddings.
- Downstream tasks: Downstream evaluation tests how suitable representations are for different problems and what semantic information they carry.The paper presents this as one of two complementary evaluation approaches.
- Downstream tasks: The study evaluates embeddings on the full set of SentEval tasks and datasets available for general-purpose sentence embeddings.The downstream tasks cover classification, entailment and relatedness, textual similarity, paraphrase detection, and caption-image retrieval.
- Linguistic probing tasks: Probing tasks classify simple linguistic properties to reveal what representations capture from a linguistic perspective.The study uses 10 probing tasks, including coordination inversion and tense detection.
4 Methods
The evaluation uses pretrained embedding models to generate sentence representations, then tests them across downstream classification, semantic similarity, retrieval, and linguistic probing tasks.
- Evaluated models: The study evaluates pretrained models and procedures for generating sentence embeddings, including ELMo, USE, Word2Vec, FastText, GloVe, p-mean, Skip-Thought, and InferSent.ELMo representations are averaged across words, using either all layers or only the top layer in separate configurations.
- Embedding construction: Sentence embeddings are produced by averaging word-level representations for Word2Vec, FastText, GloVe, and ELMo configurations.The ELMo setups use contextual features from the model's layers before averaging along the word dimension.
- Classification and probing: Downstream classification tasks use an MLP with one hidden layer of 50 neurons, Adam optimization, batch size 64, and no dropout.The Word Content probing task instead uses logistic regression because it performed consistently better there.
- Similarity evaluation: Semantic relatedness predicts scores from 0 to 5, whereas semantic textual similarity evaluates Pearson correlation between cosine similarity and human annotations.Both task families follow the evaluation scheme used in SentEval.
- Retrieval evaluation: Caption-image retrieval ranks images for captions or captions for images, using COCO and Recall@K at K = 1, 5, and 10.The setup uses 113k COCO images, each with five captions, and ResNet-101 image representations.
- Classification and probing: The evaluation includes 10 linguistic probing tasks, each targeting a different sentence-level linguistic property.Probing complements downstream evaluation by examining what linguistic information the representations encode.
5 Experimental results
Across downstream, semantic, linguistic, and retrieval evaluations, no sentence embedding method performed consistently across all tasks. ELMo bag-of-words performed broadly well, while task-specific strengths appeared for USE, InferSent, and p-mean.
- Downstream classification tasks: ELMo achieved the best results in 5 out of 9 downstream classification tasks, despite using a bag-of-words representation.The authors suggest integrating ELMo's deep representations and task-specific weighting into InferSent.
- Conclusion: Overall, the evaluated methods remained far from a universal sentence encoder with broad transfer quality.The results nevertheless identify integration of deep language-model representations as a promising direction.
- Semantic relatedness and textual similarity tasks: USE with a Transformer achieved excellent results on almost all semantic relatedness and textual similarity tasks, except SICK-R, where InferSent performed better.These task-specific differences are reported using Pearson correlation coefficients.
- Linguistic probing tasks: ELMo performed strongly across linguistic probing tasks and exceeded other methods by a large margin on BShift.The authors associate this advantage with the language-model objective's sensitivity to token-order changes.
- Linguistic probing tasks: ELMo bag-of-words achieved the best SentLen result while also performing well on downstream tasks, suggesting that its layered representation captures both superficial and deeper linguistic properties.The authors compare this multi-level aggregation to hypercolumns in computer vision and leave layer-wise probing for future work.
- Information retrieval tasks: InferSent achieved excellent results across all three recall evaluations and both caption-retrieval and image-retrieval tasks on Microsoft COCO.Table 9 reports Recall at 1, 5, and 10, together with the median.
6 Discussion
The paper comprehensively evaluates sentence embeddings through downstream transfer and linguistic probing, finding strong performance from contextual word embeddings combined with bag-of-words. However, no method performs consistently across all tasks, and entailment-trained encoders remain narrow for broad transfer.
- The evaluation covers bag-of-words baselines and supervised or self-supervised encoder architectures across inductive-transfer and linguistic-probing tasks.
- Linguistic probing tasks help reveal which relationships and properties are captured at different representation levels, especially in deep representations such as ELMo.
- No method performs consistently across all tasks; performance is linked mostly to similarity between each downstream task and the technique’s training task.
- Encoders trained on particular datasets such as entailment do not perform well across a broad set of downstream tasks, suggesting they model a narrow range of transferable information.
A.1 Supplemental Results
The supplemental results report evaluations using Logistic Regression for both linguistic probing and downstream classification tasks, with test-set accuracy as the metric.
- Table 10 reports linguistic probing-task results using Logistic Regression.The table values are test-set accuracies.
- Table 11 reports downstream-task results using Logistic Regression.The table values are test-set accuracies.