Source-linked AI summary
Universal Sentence Encoder
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, Ray Kurzweil
TL;DR
The paper addresses limited supervised data and resource constraints in NLP transfer learning. It introduces transformer and DAN sentence encoders, then evaluates their transfer performance and engineering trade-offs. Sentence-level transfer generally outperforms word-level transfer and remains effective with little task-specific data, while the models are publicly released.
Problem
Limited supervised training data and the cost of annotation constrain deep-learning approaches for many NLP tasks.
Method
The paper develops transformer and deep averaging network sentence encoders for transfer learning, comparing accuracy, resource use, and performance across training-data quantities.
Results
Sentence-level transfer tends to outperform word-level transfer, performs especially well with limited task training data, and achieves the best overall performance when combined with word-level transfer.
Takeaways & Limitations
Transfer learning is most helpful when transfer-task training data is limited, and encoder choice should reflect accuracy, memory, and compute trade-offs.
Takeaways & Limitations
Researchers and developers are encouraged to independently verify whether biases in the overall model or its components affect their use case.
Abstract
from arXiv · showhide
We present models for encoding sentences into embedding vectors that specifically target transfer learning to other NLP tasks. The models are efficient and result in accurate performance on diverse transfer tasks. Two variants of the encoding models allow for trade-offs between accuracy and compute resources. For both variants, we investigate and report the relationship between model complexity, resource consumption, the availability of transfer task training data, and task performance. Comparisons are made with baselines that use word level transfer learning via pretrained word embeddings as well as baselines do not use any transfer learning. We find that transfer learning using sentence embeddings tends to outperform word level transfer. With transfer learning via sentence embeddings, we observe surprisingly good performance with minimal amounts of supervised training data for a transfer task. We obtain encouraging results on Word Embedding Association Tests (WEAT) targeted at detecting model bias. Our pre-trained sentence encoding models are made freely available for download and on TF Hub.
1 Introduction
The paper addresses scarce supervised data in NLP by presenting sentence-embedding models for transfer learning, while examining task-data and engineering trade-offs.
- Motivation: Limited supervised training data challenges data-hungry deep learning methods across many NLP tasks.The high cost of annotation means very large supervised datasets are usually unavailable.
- Contribution: The paper presents two sentence-embedding models that transfer well to multiple NLP tasks.The models are also evaluated with varying amounts of transfer-task training data.
- Engineering trade-offs: Sentence-embedding transfer is studied alongside model complexity, memory requirements, and CPU/GPU compute time.Resource consumption is compared for sentences of varying lengths.
2 Model Toolkit
The toolkit provides two publicly available sentence encoders that map English strings to fixed-dimensional embeddings usable directly or within larger task models.
- Models: Two TensorFlow sentence encoders are provided: one transformer-based and one deep averaging network.Both models are available for download through TF Hub, although the transformer release is noted as forthcoming in the cited footnote.
- Interface: The models convert English strings into fixed-dimensional embedding representations.The embeddings can be used directly or incorporated into larger model graphs for specific tasks.
- Uses: Sentence embeddings can be used to compute semantic similarity scores or fine-tuned for specific tasks.Fine-tuning is performed with gradient-based updates when the encoder is included in a larger model.
3 Encoders
The encoders trade accuracy against efficiency: the transformer targets higher accuracy with greater resource use, while DAN provides efficient inference with strong classification performance.
- Transformer: The transformer targets high accuracy at the cost of greater model complexity and resource consumption.Its compute time and memory usage scale dramatically with sentence length.
- Transformer: The transformer constructs 512-dimensional sentence embeddings from context-aware word representations produced with attention.The representations are combined across word positions into a fixed-length vector.
- Training: Both encoders are trained for general-purpose transfer using multiple downstream tasks and a mixture of unsupervised and supervised data.Supervised training includes SNLI, which the authors observe improves transfer performance.
- DAN: The DAN encoder averages word and bi-gram input embeddings before passing them through a feedforward deep neural network.It also produces a 512-dimensional sentence embedding.
- DAN: DAN compute time is linear in input-sequence length and provides strong baseline performance on text-classification tasks.This is the primary efficiency advantage identified for the DAN encoder.
4 Transfer Tasks
The transfer evaluation covers sentiment, subjectivity, opinion polarity, question classification, semantic similarity, and model-bias characterization across established datasets and tests.
- Dataset overview: Table 1 summarizes the samples in each evaluation set, including dev and training sizes when available.The cited preparation of several datasets follows Conneau et al. (2017).
- Evaluation tasks: The evaluation set includes sentiment and subjectivity tasks drawn from movie reviews, customer reviews, and plot summaries.MR, CR, and SUBJ represent these task types.
- Evaluation tasks: The evaluation also includes opinion polarity, fine-grained question classification, and binary phrase-level sentiment classification.These correspond to MPQA, TREC, and SST, respectively.
- Evaluation tasks: STS Benchmark measures semantic textual similarity between sentence pairs using Pearson correlation with human judgments.It supplies a semantic-similarity transfer task distinct from the classification datasets.
- Bias evaluation: WEAT uses psychology-derived word pairs to characterize model bias.The paper includes WEAT data alongside transfer-task evaluation sets.
5 Transfer Learning Models
The transfer models use sentence embeddings directly or combine them with word-level representations for downstream tasks. Sentence-pair similarity is computed from embedding vectors using angular distance, while classification uses task-specific layers.
- Sentence classification tasks feed transformer or DAN sentence-encoder outputs into a task-specific DNN.
- Pairwise semantic similarity directly compares encoder outputs by converting cosine similarity into angular distance.
- Baselines contrast sentence-level transfer with word-level transfer from pretrained word2vec embeddings and with models using no transfer learning.
- Combined Transfer Models: Combined models concatenate sentence- and word-level representations before the transfer-task classification layers.
- Angular distance performs better on average than raw cosine similarity for the evaluated similarity comparisons.
- Table 2 reports accuracy for evaluations except STS Benchmark, where Pearson correlation with human judgments is reported.
6 Experiments
The experiments examine how transfer performance changes with training-data availability and model choice, while also evaluating engineering trade-offs and learned associations related to bias.
- Model hyperparameters are tuned with Vizier and manual tuning, using dev sets, cross-validation, or evaluation test data depending on availability.
- The study varies target-task training-data amounts to compare performance with and without transfer learning.
- Transformer and DAN encoders are compared to expose trade-offs between model complexity and the data needed to reach a desired task accuracy.
- The encoding models are evaluated on WEAT word lists and compared with prior GloVe-based findings on benign and potentially undesirable associations.
7 Results
Sentence-level transfer generally matches or exceeds word-level transfer, with especially strong results when target-task data are scarce. The models also show task, bias, and resource trade-offs that inform model selection.
- Sentence-level transfer generally outperforms word-level transfer, while the best performance on most tasks comes from combining both types of representations.
- The transformer encoder usually performs as well as or better than DAN, although DAN matches or exceeds it on some tasks.
- As training-set size increases, models without transfer learning approach the performance of models using transfer learning.
- Using only 1,000 SST training examples, USE T achieves performance rivaling many models trained on the full 67.3k-example set.
- The DAN encoder reproduces human associations for flowers versus insects and pleasantness versus unpleasantness, but shows weaker associations than GloVe for ageism, racism, and sexism probes.
- Discussion: Model selection should consider memory, compute, and resource requirements when task performance is close.
- Discussion: Researchers and developers are encouraged to independently verify whether biases in the overall model or its components affect their use case.
8 Resource Usage
The transformer and DAN encoders differ sharply in how compute and memory scale with sentence length. DAN remains efficient as sentences grow, while the transformer incurs increasing resource costs.
- Compute Usage: O(n^2) transformer compute and O(n) DAN compute create increasingly different costs as sentence length grows.For short sentences, transformer encoding is only moderately slower, but its compute time increases noticeably with sentence length while DAN remains nearly constant.
- Memory Usage: O(n^2) transformer memory scaling contrasts with constant DAN memory in sentence length.Transformer memory increases quickly as sentences lengthen, whereas DAN memory remains constant.
- Resource Usage: Figure 2 compares resource usage for USE D and USE T across batch sizes and sentence lengths.
- Memory Usage: Short sequences require nearly half as much memory for the transformer as for DAN.DAN memory is dominated by parameters storing unigram and bigram embeddings, while transformer memory stores only unigram embeddings.
9 Conclusion
The universal encoders provide strong sentence-level transfer across NLP tasks and outperform word-level transfer alone. Their usefulness is greatest when transfer-task training data is limited, with model choice balancing accuracy and complexity.
- Conclusion: Sentence-level embeddings from both transformer and DAN encoders surpass transfer learning using word-level embeddings alone.
- Conclusion: Models combining sentence- and word-level transfer achieve the best overall performance.
- Conclusion: Transfer learning is most helpful when limited training data is available for the transfer task.
- Conclusion: The two encoding models trade off accuracy against model complexity when selecting an encoder for an application.
- Conclusion: The pre-trained encoding models are made publicly available for research and applications that benefit from better natural-language understanding.