Source-linked AI summary
SentEval: An Evaluation Toolkit for Universal Sentence Representations
Alexis Conneau, Douwe Kiela
TL;DR
Universal sentence representations need evaluations that are broad, transferable, and comparable, but existing practices use fragmented pipelines and inconsistent settings. SentEval centralizes a consensus-based task suite with shared evaluation procedures and accessible data tooling. It is intended to make sentence-representation research fairer, less cumbersome, and more centralized.
Problem
Evaluation of general-purpose sentence embeddings is problematic because studies use varied intrinsic and downstream tasks and inconsistent evaluation pipelines, limiting comparability.
Method
SentEval provides a centralized, consensus-based suite of classification, inference, similarity, retrieval, and related transfer evaluations with common hyperparameters and accessible preprocessing tools.
Results
SentEval delivers a fair, straightforward, and centralized toolkit for evaluating sentence representations through a simple encoder interface and downloadable evaluation datasets.
Takeaways & Limitations
The toolkit is intended to help the community publish fully comparable results while the consensus on appropriate sentence-embedding evaluations evolves.
Takeaways & Limitations
The evaluation suite excludes the SMT subtask because of licensing issues.
Abstract
from arXiv · showhide
We introduce SentEval, a toolkit for evaluating the quality of universal sentence representations. SentEval encompasses a variety of tasks, including binary and multi-class classification, natural language inference and sentence similarity. The set of tasks was selected based on what appears to be the community consensus regarding the appropriate evaluations for universal sentence representations. The toolkit comes with scripts to download and preprocess datasets, and an easy interface to evaluate sentence encoders. The aim is to provide a fairer, less cumbersome and more centralized way for evaluating sentence representations.
1. Introduction
Universal sentence representations are sought for their transferability across NLP tasks, but evaluating them remains fragmented and difficult to compare. SentEval addresses this by centralizing a consensus-based evaluation toolkit and pipeline.
- Universal sentence representations aim to capture information useful across varied systems and tasks rather than optimize for one specific task.
- Sentence embeddings can improve generalization by composing unseen word combinations and encoding grammatical constructions absent from task-specific training data.
- Evaluation has spanned intrinsic similarity judgments and downstream tasks, with community consensus emerging around task sets used by influential sentence-encoder studies.
- Separate evaluation pipelines use inconsistent preprocessing, architectures, and hyperparameters, making results difficult to compare on often-small datasets.
- SentEval provides an easier way to evaluate universal sentence encoders on a large, consensus-based set of tasks.
2. Aims
SentEval aims to make research on universal sentence representations fairer, less cumbersome, and more centralized through shared evaluations, standardized procedures, and accessible tooling.
- SentEval provides one central set of evaluations based on apparent community consensus.
- It uses a common pipeline with fixed standard hyperparameters, except those tuned on validation sets, to reduce discrepancies in reported results.
- A straightforward Python interface and dataset download and preprocessing scripts make the toolkit accessible to users.
- SentEval includes example models such as a simple bag-of-words model, which may also support extrinsic evaluation of word embeddings.
3. Evaluations
SentEval evaluates sentence representations across transfer tasks spanning classification, entailment, semantic relatedness, similarity, paraphrase detection, and image-caption retrieval. The toolkit combines learned classifiers or regressors with direct cosine-based comparisons, using standardized datasets and metrics.
- Binary and multi-class classification: The evaluation suite covers binary and multi-class sentence classification tasks across sentiment, question type, product reviews, subjectivity, and opinion polarity.
- Entailment and semantic relatedness: It includes SICK entailment, SICK-R and STS Benchmark semantic relatedness, and SNLI natural language inference with contradiction, neutral, and entailment labels.
- Semantic Textual Similarity: Semantic Textual Similarity evaluates how cosine distance between sentence embeddings correlates with human similarity scores from 0 to 5.
- Paraphrase detection: Paraphrase detection uses MRPC sentence pairs annotated for paraphrase or semantic equivalence, with a two-class classifier predicting paraphrase status.
- The evaluation scope excludes the SMT subtask because of licensing issues.
- Caption-Image retrieval: Caption-image retrieval ranks images for captions or captions for images, measuring Recall@K and median rank over standardized splits.
4. Usage and Requirements
SentEval evaluates sentence encoders through a simple interface that supports transfer tasks, customizable classifiers, preprocessing, and CPU-compatible execution. Its configuration includes data paths, reproducibility settings, batching, validation, and training parameters.
- Usage: SentEval evaluates encoders by passing sentence embeddings to transfer-task classifiers or by computing cosine similarity for semantic textual similarity.Classification and regression tasks learn on top of representations, while STS tasks directly compare representation pairs.
- Interface: Users provide a batcher that converts input sentences into embeddings, and an optional prepare function can perform dataset-level preprocessing.The batcher may wrap models implemented in frameworks such as PyTorch, TensorFlow, Theano, or DyNet.
- Configuration: The evaluation script configures a data path, random seed, batch size, and selected transfer tasks before calling SentEval’s evaluation method.The task path is required, while the seed supports reproducibility and the batcher receives length-sorted minibatches.
- Classifiers: For repeated evaluations, SentEval provides a faster prototype configuration that uses RMSProp, batch size 128, tenacity 3, and epoch size 2, with slightly worse results.Additional parameters can be passed through to the prepare and batcher functions.
- Requirements: A data script downloads, unpacks, tokenizes with MOSES, and converts datasets to UTF-8 before the task path is configured.SentEval requires Python, NumPy, SciPy, recent PyTorch, and scikit-learn; CPU execution is supported where possible.
5. Baselines
SentEval evaluates several pretrained sentence-representation baselines alongside supervised systems trained separately for each task. The baselines include averaged pretrained word vectors, SkipThought, and InferSent.
- Supervised comparisons: Table 3 also reports state-of-the-art systems whose encoders and classifiers are trained directly on each task without transfer.The supervised results provide a no-transfer comparison against the pretrained representations.
- Transfer baselines: The baseline set includes continuous bag-of-words embeddings formed by averaging fastText or GloVe word vectors trained on CommonCrawl.Both Logistic Regression and MLP classifiers are reported for the bag-of-words representations.
- Transfer baselines: SkipThought and InferSent vectors are included as pretrained universal sentence-representation baselines.These methods are evaluated in the transfer setting rather than being retrained on each target task.
- Classifier comparison: Increasing classifier complexity does not improve methods with already large embedding sizes such as InferSent and SkipThought.The reported embedding sizes are 4096 for InferSent and 4800 for SkipThought.
6. Conclusion
The paper presents SentEval as a fair, centralized toolkit intended to make sentence-representation evaluation easier and results more comparable. It also identifies evolving task coverage and linguistic-property probes as future directions.
- Conclusion: SentEval centralizes sentence-representation evaluation through a simple Python interface and scripts that download the required datasets.The authors describe the toolkit as fair and straightforward.
- Conclusion: The toolkit is intended to support fully comparable results across research papers.The authors hope the community will adopt SentEval for this purpose.
- Future work: Future work includes adding tasks as consensus about appropriate sentence-embedding evaluations evolves.The paper specifically points to tasks probing linguistic properties of sentence embeddings.
- Evaluation scope: Semantic textual similarity evaluation reports Pearson correlations x100 averaged across STS’12 through STS’16 subtasks.Table 4 includes Charagram-phrase, PP-Proj, and Tree-LSTM results identified by the caption.